Overfitting Explained: Why Training Accuracy Lies

Grow a decision tree on 233 penguins with nothing stopping it: Min Node Size 1, Max Depth 12, Prune (CV) off. It reaches 26 leaves and gets every training bird right: training error 0.0%. Now score the 100 birds the Test Set control held out, which played no part in growing it. Test error is 11.0%. The model didn't get worse between those two numbers. The first one was never an estimate of anything.

Watch the gap open

Keep everything else fixed and step Max Depth from 1 to 7:

Training error falls every time the tree gets more room, because a deeper tree can always fit its own rows better. Test error stops improving at depth 3, and from there it moves within 11–14%. With 100 test rows and an error near 12%, one standard error is √(0.12 × 0.88 / 100) ≈ 3.2 points, so those moves are noise. Past depth 3, the extra 18 leaves bought nothing on new rows, and the training figure kept reporting progress.

Why training error is biased

The training rows chose the splits, so the fitted values lean toward their observed values. For squared-error loss this has an exact form. The expected optimism, meaning error on fresh responses at the same inputs minus training error, is (2/n) Σ Cov(ŷ_i, y_i) (Efron 1986; Hastie, Tibshirani & Friedman, ESL §7.4). A tree whose leaves hold one class each sets ŷ_i = y_i on every training row, which is about as much covariance as a model can have. Its training error measures how flexible the model is, not how good it is.

Bias, variance, and the pruned tree that lost

The textbook fix is pruning. Add a second Decision Tree with the default Tree Options: Min Node Size 5, Max Depth 6, Prune (CV) on. Stratum grows the tree, then picks a cost-complexity size by 5-fold cross-validation with the 1-SE Rule (rpart's default). On these 233 rows it prunes to 3 leaves, with training error 11.2% and test error 22.0%, twice the full tree's.

That's the other half of the trade-off. Expected test error is roughly bias² + variance + irreducible noise. (For 0–1 loss the terms interact rather than add, but the intuition survives.) The 3-leaf tree has low variance, and its training and test errors are close, but it's too coarse to separate the sexes within each species. That's bias. The 26-leaf tree has low bias and high variance: refit on a different 233 birds and you'd get a different tree. In an independent check over 100 random 70/30 splits (scikit-learn, not Stratum), the same unrestricted tree's test accuracy ranged from 79% to 96%. One test set gives one draw from that spread.

So pruning is a dial. It isn't a guarantee. The 1-SE rule prefers the smallest tree within one standard error of the best cross-validated error, and on small samples that can be too aggressive. Tree Size ▸ Minimum Error keeps a larger tree. Whichever you choose, the held-out rows decide whether it was right.

Accuracy hides an overconfident model

Add a Boosted Trees and a Random Forest card with defaults and the same Test Set, then open Analyze ▸ Data Mining ▸ Model Comparison:

The full tree gets the check mark for the lowest test error. But 11, 12 and 13 are one row apart each, well inside the noise, and the check mark isn't a significance test. Test AUC tells a clearer story: 0.890 for the full tree against about 0.945 for both ensembles. The full tree's leaves are pure, so every probability it gives is 0 or 1. Misclassification can't see that overconfidence. Ranking and log-loss can. The ensembles combine 100 trees each, so their probabilities are graded.

Boosting shows the same thing over time. Hold out 20% for validation, set Trees to 300 and open View ▸ Convergence. Training deviance falls to almost nothing. Validation deviance reaches its minimum near iteration 65 and then climbs steadily to iteration 300. Meanwhile validation misclassification is 9.0% at both points. The model gets more and more sure of the same answers, and only a proper scoring rule picks that up. Early Stopping cuts the model at the minimum.

What to take away

• Training error is a property of the fit, not a forecast. Report it next to held-out error, never instead of it.
• Validation rows make choices, so they're optimistic too. Keep a test set that's scored once, after every choice. In the companion workflow, a tree pruned on 67 validation rows showed 11.9% there and 22.4% on test.
• Compare models on the same rows, with an error bar in your head. On 100 rows, differences under about 3 points aren't evidence.
• Look past accuracy. AUC and deviance catch overconfidence that the misclassification rate rounds away.

Download Stratum

Frequently asked questions

What is overfitting?

A model has overfit when it has learned the noise in its training rows as well as the signal. The symptom is a gap: error on the rows it was fitted to is much lower than error on new rows from the same population. The size of the gap is the size of the bias.

Is an overfit model always worse than a simpler one?

No. Overfitting says the training error is wrong, not that the model is bad. Here the 26-leaf tree overfit and still beat the pruned tree on test rows. Only held-out error can tell you which to use.

Why isn't validation error enough?

Validation rows are used to make choices, such as tree size or the number of boosting iterations, so the chosen model is tuned to them. A test set scored once, after every choice, gives the unbiased estimate.

How do I see overfitting in a boosted model?

Hold out a validation set and open the Convergence view. Training deviance keeps falling, while validation deviance bottoms out and rises. Early Stopping keeps only the trees up to the minimum.

Why can accuracy and AUC disagree about which model is best?

Accuracy scores only the predicted class. AUC scores how well the predicted probabilities rank the rows. A tree with pure leaves predicts 0 or 1, so it can match an ensemble on accuracy and trail it badly on AUC.

Related: Train, Validation and Test Sets — Without Code → · The Penguin Paradox → · Decision Trees Explained →