Hyperparameter Tuning Tested: Four Classifiers, One Dataset, No Real Improvement

Image by author

Introduction: What the Experiment Tested

Hyperparameter tuning is often presented as a magic pill for machine learning. The promise sounds simple: adjust a few parameters, run a grid search for a few hours, and watch the model’s performance climb. Does that hold up in practice?

Image by author

This experiment, conducted by data scientist Nate Rosidi, put the premise to the test on Portuguese student performance data using four different classifiers and rigorous statistical validation: nested cross-validation, leakage-proof preprocessing pipelines and significance testing.

The outcome? Average performance dropped by 0.0005. Tuning made the results marginally worse, and the difference was not statistically significant. That is not a failure: in many real projects default settings work remarkably well, and knowing when to stop tuning is itself a valuable skill. The full experiment is available as a complete Jupyter notebook with all code and analysis.

The Dataset

Image by author

The data comes from the “Student Performance Analysis” project on StrataScratch. It contains records of 649 students with 30 attributes covering demographics, family background, social factors and school-related information. The objective was to predict whether a student passes the final Portuguese class (a score of 10 or higher).

An important design decision was to exclude the G1 and G2 period grades. These first- and second-period marks correlate at 0.83–0.92 with the final grade, G3; including them makes the prediction trivially easy and defeats the purpose of the experiment, which was to identify what predicts success beyond prior performance in the same course.

The pandas library was used for loading and preparing the data:

# Load and prepare data
df = pd.read_csv('student-por.csv', sep=';')

# Create pass/fail target (grade >= 10)
PASS_THRESHOLD = 10
y = (df('G3') >= PASS_THRESHOLD).astype(int)

# Exclude G1, G2, G3 to prevent data leakage
features_to_exclude = ('G1', 'G2', 'G3')
X = df.drop(columns=features_to_exclude)

The class distribution showed that 100 students failed (15.4%) while 549 passed (84.6%). Because the classes are imbalanced, the experiment optimised for F1-score rather than simple accuracy.

The Four Classifiers

Four classifiers representing different learning approaches were selected:

Image by author

Each model was first run with default parameters, then tuned via grid search with five-fold cross-validation.

A Deliberately Strict Methodology

Many tutorials demonstrate impressive tuning gains because they omit important validation steps. This experiment maintained strict standards so the findings would be reliable:

• No data leakage. All preprocessing was done inside pipelines and fitted only on training data.
• Nested cross-validation. An inner loop handled hyperparameter tuning and an outer loop handled the final evaluation.
• A proper train/test split. An 80/20 stratified split, with the test set kept untouched until the end.
• Statistical validation. McNemar’s test was applied to verify whether performance differences were statistically significant.
• Metric selection. F1-score was prioritised over accuracy because of the class imbalance.

Image by author

The pipeline structure was as follows:

# Preprocessing pipeline - fit only on training folds
numeric_transformer = Pipeline((
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
))

categorical_transformer = Pipeline((
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('onehot', OneHotEncoder(handle_unknown='ignore'))
))

# Combine transformers
from sklearn.compose import ColumnTransformer
preprocessor = ColumnTransformer(transformers=(
    ('num', numeric_transformer, X.select_dtypes(include=('int64', 'float64')).columns),
    ('cat', categorical_transformer, X.select_dtypes(include=('object')).columns)
))

# Full pipeline with model
pipeline = Pipeline((
    ('preprocessor', preprocessor),
    ('classifier', model)
))

The Results

After the tuning process completed, the results were striking:

The average improvement across all models was −0.0005. Three of the four models performed slightly worse after tuning. XGBoost showed an improvement of about 1%, which looked promising until the statistical tests were applied: on the hold-out test set, none of the models showed a statistically significant difference.

McNemar’s test comparing the two best-performing models returned a p-value of 1.0 — no significant difference between the default and tuned versions.

Why Hyperparameter Tuning Did Not Help

Image by author

Several factors explain the result:

• Strong defaults. scikit-learn and XGBoost ship with carefully chosen default parameters, refined by maintainers over years to work well across a wide range of datasets.
• Limited signal. After removing the G1 and G2 grades to prevent leakage, the remaining features had modest predictive power — there was little signal left for hyperparameter optimisation to exploit.
• Small dataset. With only 649 samples split across training folds, grid search had too little data to reliably distinguish between parameter sets. Where collecting more real data is impractical, approaches such as synthetic data generation are one way teams work around this.
• A performance ceiling. The baseline models already scored roughly 92–93% F1; without better features or more data, headroom was limited.
• Strict methodology. When data leaks are eliminated and nested cross-validation is used, the inflated gains often seen with improper validation disappear.

Lessons for Practitioners

Image by author

• Methodology matters more than metrics. Fixing leakage and validating properly changes the outcome; impressive gains from sloppy validation vanish under scrutiny.
• Statistical validation is essential. Without McNemar’s test, the nominal 1% XGBoost improvement could have justified deploying the wrong model. Testing showed it was noise.
• Negative results are valuable. Knowing when tuning does not help saves time on future projects and signals a mature workflow.
• Defaults are underrated. For standard datasets, default hyperparameters are often sufficient; tuning every parameter from the start is rarely necessary.

Limitations and What to Watch

The findings come with caveats. This is a single experiment on one small, tabular dataset; on larger datasets, with stronger feature signal, or with models more sensitive to their settings, tuning can still deliver meaningful gains. Grid search is also only one tuning strategy — random search and Bayesian optimisation can explore parameter space more efficiently. The result is best read as a caution against expecting tuning to rescue a weak feature set, not as evidence that tuning never works.

Summary

The experiment attempted to boost model performance through thorough hyperparameter tuning, industry best practices and statistical validation across four models. The result: no statistically significant improvement.

Image by author

Rather than a failure, that is the kind of honest result that leads to better decisions in real-world projects: it shows when to stop tuning and when to shift attention to data quality, feature engineering or collecting additional samples — the same priorities that surface in applied work such as customer-call sentiment analysis with Whisper and BERTopic.

Machine learning is not about squeezing out the highest possible number by any means available; it is about building models that can be trusted. That trust comes from sound methodology, not from chasing marginal gains. One of the hardest skills in machine learning is knowing when to stop.

Image by author

Nate Rosidi is a data scientist and product strategist, an adjunct professor teaching analytics, and the founder of StrataScratch, a platform that helps data scientists prepare for interviews with real questions from top companies.