Model Selection

I'll be honest — I spent an embarrassing amount of time early in my career skipping straight to the fanciest model I could find. Someone would hand me a dataset and I'd reach for a neural net or a gradient boosting ensemble before I'd even looked at a histogram of the target variable. I'd tune hyperparameters for hours, get a number that looked impressive, and feel good about myself. Then the model would fall apart in production, and I'd have no idea why, because I'd never established what "good" even looked like for that problem. I didn't have a floor. I didn't have a ceiling. I had a number floating in space.

It took a few painful failures — and one particularly humbling incident where a linear regression outperformed my carefully tuned 500-tree ensemble — for the real lesson to sink in. Model selection isn't about picking the most powerful algorithm. It's about systematically understanding what your data needs and matching the right tool to the job. This section is that understanding, built from scratch.

Model selection is the process of choosing which algorithm, which level of complexity, and which configuration to use for a given problem. It sits at the heart of every ML workflow, and getting it right requires a blend of statistical reasoning, practical judgment, and honest self-assessment. The field has developed surprisingly principled tools for this — from information-theoretic criteria like AIC and BIC, to Bayesian search strategies, to game-theory-based explanation methods.

Before we start, a heads-up. We're going to touch on some mathematical ideas — likelihood functions, kernel density estimation, Shapley values from game theory — but you don't need any of that beforehand. We'll build each concept from the ground up when we need it.

This isn't a short journey, but I hope you'll be glad you came.

The Uncomfortable Truth

Before we get into any of the mechanics, there's something we need to confront head-on. In 1997, David Wolpert and William Macready published a theorem that shattered a comforting illusion. The No Free Lunch Theorem proves, mathematically, that if you average over every possible dataset, no learning algorithm outperforms any other. Every algorithm that excels on some problems must pay for it by being mediocre on others.

I remember my reaction the first time I understood the implications of this. It felt like finding out there's no best restaurant — that the question itself doesn't have an answer without specifying what you're hungry for. The algorithm that crushes your churn prediction might stumble on your recommendation engine. The neural net that dominates image recognition might be embarrassingly beaten by a logistic regression on your tabular fraud dataset.

But here's why this is actually liberating rather than depressing. It means the right answer to "which model should I use?" is always: "let's find out, on your data, with honest evaluation." No one can hand you a universal answer. That means the process of model selection — what we're about to build — is where the real craft lives.

The limitation of the theorem, though, is that we never actually encounter "every possible dataset." Real data has structure. Images have spatial correlations. Text has grammar. Tabular data has column relationships. The art is exploiting that structure, and different algorithms exploit different kinds of structure. That's why we need a process.

A Toy Problem Worth Caring About

Let's give ourselves something concrete to work with. Imagine we're building a model for a small real-estate startup. They want to predict whether a house will sell within 30 days of listing. We have a dataset with 5,000 rows and 12 features: square footage, number of bedrooms, neighborhood crime rate, school rating, listing price, distance to downtown, lot size, year built, garage spaces, recent renovation (yes/no), previous days on market, and season listed.

Our target is binary: sold within 30 days or not. About 35% of houses sell that fast — so this isn't a wildly imbalanced problem, but predicting the majority class ("not sold") already gets you 65% accuracy. Keep that number in your head. It's about to become very important.

This toy problem will follow us through the entire section. Every concept we build, we'll test against these 5,000 houses.

Start With the Dumbest Thing That Could Work

Here's the habit that separates people who waste months from people who ship models: always start with a baseline that requires zero intelligence. For our house-selling problem, the dumbest possible predictor is one that always says "not sold within 30 days" — because that's the majority class. It will be right 65% of the time without ever looking at a single feature.

from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, roc_auc_score

baseline = DummyClassifier(strategy="most_frequent")
baseline.fit(X_train, y_train)
y_pred = baseline.predict(X_test)
print(f"Baseline accuracy: {accuracy_score(y_test, y_pred):.3f}")
# Baseline accuracy: 0.650

That 0.650 is our floor. If we spend a week engineering features and training a complex model and get 0.660, we've learned almost nothing. If we get 0.850, we've learned something real. Without the baseline, we'd never know which of those two worlds we're in.

I cannot overstate how many times I've seen teams skip this step and spend weeks debugging a pipeline that was "performing well" — only to discover their fancy model was performing at baseline level. The model had learned nothing. The features were uninformative. A 30-second check would have revealed this on day one.

The baseline is your sanity check. It's also your floor for another purpose: it sets the denominator when you calculate how much each subsequent model is actually earning its complexity. That brings us to the uncomfortable question of when to reach for more.

Earning Your Way to Complexity

After the dummy baseline, the next step is the least complex real model you can think of. For our house classification, that's a logistic regression. It trains in under a second. It gives you a coefficient for each feature — you can look at those numbers and immediately understand what the model thinks matters. If the listing price coefficient is large and negative, the model is telling you that higher prices make quick sales less likely. That's interpretable. That's debuggable.

from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

simple_model = make_pipeline(StandardScaler(), LogisticRegression())
simple_model.fit(X_train, y_train)
y_pred = simple_model.predict(X_test)
print(f"Logistic Regression accuracy: {accuracy_score(y_test, y_pred):.3f}")
# Logistic Regression accuracy: 0.742

From 0.650 to 0.742. That's a real improvement. The model is learning something. Now the question: can we do better, and is the cost of doing better worth it?

A random forest is the natural next step. More complex, harder to interpret, but capable of capturing non-linear relationships the logistic regression misses. Then gradient boosting. Then maybe a neural net if the data is large enough to justify it.

Think of model complexity like floors in a building. The ground floor (baseline) is free. Each additional floor costs more to build, takes longer, and is harder to maintain. If most of the value is on the second floor (logistic regression), you don't need a skyscraper. I still get tripped up by this temptation — the pull toward the most sophisticated tool. It's worth resisting.

Here's the rough guide that saves a lot of wasted compute:

This isn't a law. A well-regularized neural net can work on 5K rows, and logistic regression can work beautifully on millions of rows if the relationship is actually linear. But violating this rule of thumb — throwing a transformer at 200 samples — almost always ends in tears. Our house dataset with 5,000 rows puts us squarely in the "gradient boosting sweet spot," but we earned that knowledge by trying the simpler models first.

The Knob-Turning Problem

Every model has knobs. A random forest has number of trees, max depth, minimum samples per leaf. A gradient boosting model adds learning rate and subsample ratio on top of those. A neural net has layer sizes, dropout rate, optimizer choice, weight decay. These are hyperparameters — values you set before training begins, as opposed to parameters the model learns from data during training.

The right hyperparameter values can make a shockingly large difference. I've seen the same algorithm go from mediocre to state-of-the-art with nothing but a learning rate change. The wrong learning rate can make a gradient boosting model train for hours and produce garbage. The right one can get you to 95% of optimal performance in minutes.

So how do we find good values? Three approaches, in order of sophistication. Each one's limitation motivates the next.

Grid Search and Why It Wastes Your Time

The first idea is the most intuitive: pick a set of values for each hyperparameter, try every combination, keep the best. If we're tuning our gradient boosting model for the house dataset and we want to try 4 values for n_estimators, 3 for max_depth, and 3 for learning_rate, that's 4 × 3 × 3 = 36 combinations. With 5-fold cross-validation, that's 180 model fits.

This is grid search. It's thorough in the sense that it covers every intersection of the grid. And it feels responsible — you've tried every combination, right?

But grid search has a problem that becomes obvious once you think about it. In most ML problems, one or two hyperparameters matter a lot and the rest barely matter at all. For gradient boosting, the learning rate typically dominates performance. Max depth matters somewhat. The exact number of estimators past a certain threshold? Often doesn't change much. Grid search doesn't know this. It spends equal budget exploring all axes, including the ones that don't matter.

Picture it geometrically. If you have a 9-point grid over two dimensions and only the horizontal axis matters, all 9 points give you only 3 unique values on that important axis. You've spent 6 of your 9 evaluations on duplicates along the dimension that counts. That's expensive redundancy.

Random Search: The Lazy Genius

In 2012, Bergstra and Bengio published a paper with a finding that surprised a lot of people: sampling hyperparameters at random from distributions you specify beats grid search in most practical scenarios. The paper showed that 60 random trials is at least as effective as a grid search with 10 intervals per hyperparameter — and often much better.

The reason is elegant. If only the learning rate matters much, and you run 60 random trials, you get 60 different learning rate values. Grid search with the same budget might only give you 4 or 5 unique learning rate values, because the rest of the budget is spent varying the unimportant parameters. Random search doesn't waste trials on a grid. It explores the important dimensions more thoroughly by accident.

from sklearn.model_selection import RandomizedSearchCV
from sklearn.ensemble import GradientBoostingClassifier
from scipy.stats import uniform, randint

param_distributions = {
    "n_estimators": randint(50, 500),
    "max_depth": randint(2, 10),
    "learning_rate": uniform(0.001, 0.3),
    "subsample": uniform(0.6, 0.4),
    "min_samples_leaf": randint(1, 20),
}

search = RandomizedSearchCV(
    GradientBoostingClassifier(random_state=42),
    param_distributions,
    n_iter=100,        # 100 random combinations
    cv=5,
    scoring="roc_auc",
    n_jobs=-1,
    random_state=42,
)
search.fit(X_train, y_train)
print(f"Best AUC: {search.best_score_:.4f}")
print(f"Best params: {search.best_params_}")

For our house dataset, 100 random trials takes a few minutes and consistently finds configurations that match or beat what grid search finds with thousands of evaluations. The key detail: specify distributions, not lists. Use uniform for continuous parameters and randint for integers. For parameters where small values are more interesting (like learning rate), a log-uniform distribution is even better — it samples proportionally more from the low end.

But random search has its own limitation. Each trial is independent. Trial 50 knows nothing about what trials 1 through 49 discovered. If trial 12 found that learning rates below 0.01 always perform poorly, trial 50 might still waste time trying 0.003. We're not learning from our experiments. That feels like it should bother us. It does.

Bayesian Optimization: Learning From Failures

What if, after each trial, we built a model of how hyperparameters relate to performance — and used that model to decide where to search next? That's Bayesian optimization. After evaluating a configuration, it updates a probabilistic model (called a surrogate model) of the objective function. Then it uses that surrogate to pick the next configuration to try — spending more time in regions that look promising and less time in regions that have consistently underperformed.

The most popular flavor in practice is called TPE — Tree-structured Parzen Estimators. It works by splitting all the trials seen so far into two groups: the good ones (top 20% or so) and the bad ones. Then it fits a density estimate over each group separately. A candidate configuration is promising if it looks a lot like the good trials and unlike the bad ones. Mathematically, it picks the candidate that maximizes the ratio l(x)/g(x), where l(x) is the density of good trials and g(x) is the density of bad ones.

I'll be honest — when I first learned that TPE models P(x|y) instead of P(y|x) (that is, it models "given a good outcome, what do the parameters look like?" rather than "given these parameters, what outcome do we expect?"), it felt backward. But it turns out this inversion is what makes TPE handle categorical parameters, conditional spaces, and high dimensions so well. The traditional Gaussian Process approach to Bayesian optimization works beautifully in low dimensions with continuous parameters, but starts to struggle in the messy, mixed-type search spaces that real hyperparameter tuning involves.

The practical payoff: Bayesian optimization typically needs 2-5x fewer trials than random search to find configurations of equivalent quality. On our house dataset, where each gradient boosting cross-validation takes a few seconds, the time savings is modest. But on a deep learning problem where each trial takes an hour? Cutting 200 trials down to 50 saves days.

Optuna: What Practitioners Actually Use

Optuna is the tool that most production ML teams reach for when they need hyperparameter optimization. It implements TPE by default, has a clean Pythonic API, and supports a feature that makes it genuinely different from other tuning tools: pruning.

Pruning is early stopping for hyperparameter trials. As a model trains, it periodically reports intermediate performance (say, validation loss after each epoch). Optuna's MedianPruner compares this intermediate performance to the median of all completed trials at the same step. If the current trial is performing below the median, Optuna kills it. No sense waiting 50 more epochs to confirm what's already looking bad.

import optuna
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import GradientBoostingClassifier

def objective(trial):
    params = {
        "n_estimators": trial.suggest_int("n_estimators", 50, 500),
        "max_depth": trial.suggest_int("max_depth", 2, 10),
        "learning_rate": trial.suggest_float("learning_rate", 1e-3, 0.3, log=True),
        "subsample": trial.suggest_float("subsample", 0.6, 1.0),
        "min_samples_leaf": trial.suggest_int("min_samples_leaf", 1, 20),
    }
    model = GradientBoostingClassifier(**params, random_state=42)
    scores = cross_val_score(model, X_train, y_train, cv=5, scoring="roc_auc")
    return scores.mean()

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=100, show_progress_bar=True)

print(f"Best AUC: {study.best_trial.value:.4f}")
print(f"Best params: {study.best_trial.params}")

After the study finishes, Optuna gives you something that I find more valuable than the best parameters themselves: plot_param_importances(study). This tells you which hyperparameters actually mattered. If learning rate accounts for 80% of the performance variance and subsample accounts for 2%, you know where to focus future effort — and you know which knobs you can stop worrying about.

Rest Stop

If you've made it this far, congratulations. You can stop here if you want.

You now have a working mental model for model selection: start with a dumb baseline, earn complexity step by step, match model capacity to data size, and tune hyperparameters with random search or Bayesian optimization instead of grid search. That's genuinely enough to handle most real-world ML projects competently.

The short version for those who want to bail: always baseline → simple → complex. Use Optuna for tuning. Ship the thing that works. There. You're 80% of the way there.

But there are some powerful ideas ahead that we haven't touched — principled ways to compare models (AIC and BIC), diagnostic tools that tell you exactly what's wrong with your model (learning curves), a rigorous way to avoid fooling yourself about performance (nested cross-validation), and a method from game theory that lets you explain any model's predictions (SHAP). If the discomfort of not knowing what's underneath is nagging at you, read on.

Information Criteria: AIC and BIC

Cross-validation tells you how well a model generalizes, but it doesn't directly tell you whether the complexity you've added is worth it. Two models might have similar cross-validation scores, but one uses 3 parameters and the other uses 30. Are those extra 27 parameters earning their keep, or are they memorizing noise?

Information criteria answer this question with a formula. The Akaike Information Criterion (AIC) is defined as AIC = 2k − 2 ln(L), where k is the number of parameters and L is the maximized likelihood. The first term penalizes complexity. The second rewards goodness of fit. Lower AIC is better.

Let's trace through a tiny example with our house data. Suppose we fit two logistic regression models — one with all 12 features (k = 13 including intercept, log-likelihood = −1,420) and one with only 4 features (k = 5, log-likelihood = −1,445). The full model fits the data better (higher likelihood), but uses more parameters.

AIC for the full model: 2(13) − 2(−1,420) = 26 + 2,840 = 2,866. AIC for the reduced model: 2(5) − 2(−1,445) = 10 + 2,890 = 2,900. The full model wins by 34 points. Since differences greater than 10 constitute strong evidence, those extra 8 features are genuinely earning their keep here.

The Bayesian Information Criterion (BIC) uses a harsher penalty that grows with sample size: BIC = ln(n)·k − 2 ln(L). With n = 5,000, ln(5000) ≈ 8.5, so each parameter costs 8.5 instead of 2. BIC punishes complexity more aggressively, especially on large datasets. It tends to prefer simpler models — which makes it useful when you care more about finding the true underlying model than about squeezing out the last fraction of predictive accuracy.

The practical rule of thumb: use AIC when your goal is prediction (you'll accept some extra parameters if they help). Use BIC when your goal is inference or understanding (you want the simplest model that captures the real signal). When they agree, you can be fairly confident. When they disagree, AIC is saying "the extra complexity helps predict" and BIC is saying "but it might not reflect the true structure." Both are valuable perspectives.

A limitation worth flagging: AIC and BIC assume you're fitting models with maximum likelihood estimation. For tree-based models like random forests or gradient boosting, they don't apply directly. For those, cross-validation remains your primary tool.

Learning Curves: Your Model's Diary

There's a diagnostic tool that I wish someone had showed me earlier in my career. A learning curve plots your model's training score and validation score as a function of the number of training samples. It's like an X-ray of your model's health. Two minutes of plotting can save you days of confused debugging.

Here's how to read one. If both the training and validation scores are low and they've converged close together, your model is underfitting. It's too simple to capture the patterns in the data. Adding more data won't help — you need a more complex model or better features. Back to our building analogy: you're on a floor that's too low, and no amount of furniture will change that.

If the training score is high but the validation score is much lower — a visible gap between the two curves — your model is overfitting. It's memorized the training data but can't generalize. The fix is either more data (watch if the validation curve is still climbing as you add samples), more regularization, or a simpler model.

from sklearn.model_selection import learning_curve
import numpy as np

train_sizes, train_scores, val_scores = learning_curve(
    GradientBoostingClassifier(n_estimators=200, max_depth=4, random_state=42),
    X_train, y_train,
    cv=5,
    scoring="roc_auc",
    train_sizes=np.linspace(0.1, 1.0, 10),
    n_jobs=-1,
)

# Plot train_scores.mean(axis=1) and val_scores.mean(axis=1)
# against train_sizes to see the learning curve

For our house dataset, if the learning curve shows the validation score still climbing when we use all 5,000 training samples, that's a strong signal: more data would help. If it's plateaued, more data won't buy us much — we need better features or a different model. I still check learning curves on every project. It takes two minutes and has saved me from going down the wrong path more times than I can count.

Nested Cross-Validation: The Honest Referee

Here's a subtle trap that catches a lot of people, including past-me. You use cross-validation to tune hyperparameters (the inner loop), then report the best cross-validation score as your model's performance. Feels natural, right? But there's a problem: you've selected hyperparameters that performed best on those specific folds. You've optimized for the validation data. Your reported performance is biased upward — sometimes by a lot.

Nested cross-validation fixes this with two loops. The inner loop handles hyperparameter tuning. It uses only the training portion of each outer fold. The outer loop evaluates performance on held-out data that the inner loop never saw and never influenced.

from sklearn.model_selection import cross_val_score, GridSearchCV

inner_cv = GridSearchCV(
    GradientBoostingClassifier(random_state=42),
    param_grid={"max_depth": [3, 5, 7], "learning_rate": [0.01, 0.1]},
    cv=3,
    scoring="roc_auc",
)

# Outer loop: evaluates the entire tuning process on truly held-out data
outer_scores = cross_val_score(inner_cv, X, y, cv=5, scoring="roc_auc")
print(f"Nested CV AUC: {outer_scores.mean():.4f} ± {outer_scores.std():.4f}")

The outer scores are an honest estimate of how your entire pipeline — including the hyperparameter selection process — will perform on new data. The inner CV score is always more optimistic. The gap between inner and outer tells you how much of your performance was real versus how much was selection bias. I've seen this gap be as large as 5 percentage points on small datasets. That's the difference between "this model works" and "this model doesn't actually work but the validation made it look like it does."

The limitation: nested CV is expensive. With 5 outer folds and 3 inner folds, you're fitting 15x as many models. For quick iteration, regular cross-validation is fine. But for your final reported numbers — the ones that go in the paper, the ones you tell your manager, the ones that justify the production deployment — nested CV is the honest answer.

Explaining Yourself: SHAP

You've selected a gradient boosting model for the house dataset. It gets 0.87 AUC. Great. Your startup's CEO asks: "Why did the model predict that the house on Maple Street would sell slowly?" If your answer is "the model thinks so," you've lost their trust.

SHAP — SHapley Additive exPlanations — gives you a real answer. It's based on an idea from cooperative game theory called Shapley values, introduced by Lloyd Shapley in 1953. The core question Shapley values answer: if a group of players cooperated to produce an outcome, how should the credit be fairly distributed among them?

In our context, the "players" are the features, and the "outcome" is the model's prediction. For the Maple Street house, SHAP might say: the listing price pushed the prediction toward "slow sale" by +0.15, the school rating pushed toward "fast sale" by −0.08, distance to downtown pushed toward "slow sale" by +0.06, and so on. The sum of all these contributions, plus a baseline (the average prediction), equals the model's prediction for that specific house.

That additivity property — baseline + sum of SHAP values = prediction — is what makes SHAP so powerful. It's not an approximation. It's an exact decomposition. Every fraction of the prediction is accounted for.

import shap

model = GradientBoostingClassifier(n_estimators=200, random_state=42)
model.fit(X_train, y_train)

explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)

# Global view: which features matter most across all houses?
shap.summary_plot(shap_values, X_test, feature_names=feature_names)

# Local view: why THIS prediction for the Maple Street house?
shap.force_plot(
    explainer.expected_value, shap_values[42],
    X_test.iloc[42], feature_names=feature_names
)

The summary_plot is the single most information-dense visualization in ML explainability. Each row is a feature. Each dot is a data point. Horizontal position shows how much that feature pushed the prediction. Color shows the feature value (red = high, blue = low). You can immediately see patterns: high listing prices push toward slow sales, high school ratings push toward fast sales. One plot tells you the story of your entire model.

Computing exact Shapley values is exponentially expensive — you'd need to evaluate the model on every possible subset of features. SHAP makes this practical through clever approximations. TreeSHAP exploits the structure of tree-based models to compute exact values in polynomial time. KernelSHAP works on any model but uses a sampling approach that's slower. For our gradient boosting house model, TreeSHAP runs in seconds. For a neural net, KernelSHAP might take minutes per prediction.

A lighter alternative when you don't need per-prediction explanations is permutation importance: shuffle one feature's values, measure how much performance drops. Big drop means important feature. It's model-agnostic and useful as a quick sanity check. But it can't tell you why a specific house was predicted the way it was — only SHAP does that.

AutoML: Letting the Machine Choose

At some point, a reasonable question arises: if model selection is about trying different algorithms and tuning their hyperparameters, can we automate the whole thing?

That's exactly what AutoML frameworks do. Auto-sklearn combines Bayesian optimization with meta-learning — it has a memory of which algorithms worked well on similar datasets in the past, and uses that to warm-start the search. It also automatically builds ensembles of the top-performing models. FLAML (from Microsoft) takes a different philosophy: be fast and lightweight, find a good-enough model quickly rather than spending hours chasing the last fraction of a percent.

I'm still developing my intuition for when AutoML is worth reaching for versus doing the selection manually. My current thinking: AutoML is excellent for establishing a strong baseline quickly, for exploring model types you might not have thought to try, and for situations where you need a working model fast and don't have deep ML expertise on the team. It's less useful when you need to understand exactly why a particular model was chosen, when you have unusual constraints (custom loss functions, deployment size limits, latency requirements), or when the data requires domain-specific preprocessing that AutoML frameworks don't know about.

The biggest trap with AutoML is treating it as a black box. "Auto-sklearn picked a stacking ensemble of 7 models" — great, but do you understand why? Can you explain it? Can you debug it when it fails in production? AutoML automates the tedious parts, but it doesn't replace understanding.

When Simpler Models Win

There's an uncomfortable truth that doesn't get enough airtime: simpler models win more often than the ML community's obsession with state-of-the-art results would suggest.

For tabular data — the kind in databases, CSV files, business analytics — gradient boosting (XGBoost, LightGBM, CatBoost) has dominated for years. This isn't controversial. Kaggle competitions, academic benchmarks, production systems — the evidence is overwhelming. Deep learning on tabular data is an active research area with promising results (TabNet, FT-Transformer), but the practical consensus: gradient boosting first, neural nets only if you have a specific reason like massive scale, multimodal inputs, or transfer learning from pretrained models.

Under a few thousand rows, even gradient boosting can struggle. Linear models and small ensembles become your best options. A logistic regression with well-engineered features will frequently beat a neural net that's starving for data. I keep having to relearn this lesson.

When latency matters — real-time bidding, high-frequency trading, edge devices — a logistic regression that runs in microseconds beats a 500-tree ensemble every time. The ensemble might be more accurate, but a single matrix multiplication fits in an L1 cache and a forest of trees doesn't.

When a regulator says "explain this decision," a linear model can answer in one sentence: "Feature A contributed +0.3, Feature B contributed −0.2, baseline is 0.1, total is 0.2, above the threshold of 0.15." Explaining a neural net's decision requires SHAP, which requires explaining SHAP to the regulator, which requires the regulator to trust a game-theory-based explanation of a model they already didn't trust. Each layer of explanation adds a layer of skepticism.

And when the underlying relationship is approximately linear, a nonlinear model adds variance without reducing bias. It's like using a rocket launcher to open a jar. Check for nonlinearity before reaching for nonlinear models. A scatter plot takes 10 seconds and can save you days.

The Workflow That Actually Ships

Everything we've built comes together in a sequence that I've used on dozens of projects. It works in production, not in theory.

First, establish a baseline. Fit a DummyClassifier or DummyRegressor. This takes 30 seconds and gives you the floor. For our house dataset, it's 0.650 accuracy. If your complex model can't beat this, your features are uninformative — fix the data, not the model.

Second, try simple models. Logistic regression, linear regression, a single decision tree. They train in under a second, they're interpretable, and they're often surprisingly competitive. Our logistic regression got 0.742. That's our first real benchmark.

Third, move to complex models. Random forest, gradient boosting, maybe an SVM. Compare their cross-validation scores against the simple models. If the improvement is marginal — say 0.5% AUC — think hard about whether the complexity is worth it.

Fourth, validate honestly. Run nested cross-validation on your top 2-3 candidates. Report mean and standard deviation. If two models are within one standard deviation of each other, prefer the simpler one. Use a paired t-test or Wilcoxon signed-rank test if you need statistical confirmation that one is truly better.

Fifth, check deployment constraints. Does the model need to predict in under 10ms? Fit in 500MB of RAM? Run on a phone? Does someone need to sign off on every prediction? These constraints can eliminate your best-performing model entirely. Better to discover this now than after three months of productionizing.

Sixth, decide and document. Write down which models you tried, their scores, why you picked the one you picked, and what tradeoffs you accepted. "Selected LightGBM (AUC 0.874 ± 0.012) over logistic regression (AUC 0.831 ± 0.015): 4-point improvement justified added complexity. LR remains fallback if interpretability requirements change." Future-you will thank you.

Wrap-Up

If you're still with me, thank you. I hope it was worth it.

We started with an uncomfortable mathematical truth — that no model is universally best — and used it as motivation to build a disciplined process. We established baselines, earned complexity step by step, learned why grid search wastes your budget and how random search and Bayesian optimization spend it more wisely. We picked up principled tools: AIC and BIC for judging whether complexity is earning its keep, learning curves for diagnosing what's wrong, nested cross-validation for honest performance estimates, and SHAP for explaining predictions to the people who need to trust them. And we threaded all of this through 5,000 houses that needed predicting.

My hope is that the next time someone hands you a dataset and asks "which model should we use?", instead of reaching for the fanciest thing you know, you'll start with a baseline, work your way up systematically, tune with intelligence rather than brute force, and ship the thing that actually works — with a clear explanation of why you chose it.

Resources

• Bergstra & Bengio (2012), "Random Search for Hyper-Parameter Optimization" — The paper that proved random search beats grid search. Wildly influential and a satisfying read. PDF
• Optuna documentation — Clean, well-written docs with excellent tutorials on pruning and visualization. This is the tool you'll actually use. optuna.readthedocs.io
• Lundberg & Lee (2017), "A Unified Approach to Interpreting Model Predictions" — The original SHAP paper. Dense but insightful. The connection to Shapley values is beautifully constructed. NeurIPS
• Burnham & Anderson (2002), "Model Selection and Multimodel Inference" — The definitive guide to AIC, BIC, and information-theoretic model selection. If you want to understand why these criteria work, start here.
• Hastie, Tibshirani & Friedman, "The Elements of Statistical Learning" (Ch. 7) — The gold standard treatment of model assessment and selection. Free PDF available. Unforgettable once you work through it.
• scikit-learn User Guide: Model Selection — Practical, code-first documentation covering cross-validation, grid search, and learning curves. scikit-learn.org