Train, Validation and Test Sets — Without Code

Q: How large should the validation and test sets be?

Large enough that the error estimate is precise enough for your decision. The standard error of an error rate p on n test rows is about the square root of p(1 − p)/n, so 100 rows at 12% error gives roughly ±3 points. In Stratum each held-out set can take up to 50% of the rows, and the two together up to 80%, in whole percent.

Q: Do two models in Stratum hold out the same rows?

Yes, when they predict the same response with the same Validation and Test Set fractions. Each row's role comes from its own seeded draw, not from a shuffle of the rows a model happens to use, so changing predictors or the filter does not reshuffle anyone. Model Comparison checks this and only marks a best model when every model held out the same rows.

Q: Can I bring my own split column?

Yes. Choose it as the Validation Column. It can hold 0, 1 and 2, or the words Training, Validation and Test. A row with any other value takes no part in the fit. Save as Column writes exactly this kind of column from the current split.

One split, three models, one honest comparison. Here's how to hold out validation and test rows in Stratum and find out which model actually generalizes.

Follow along: penguins.csv from palmerpenguins (Horst, Hill & Gorman): 344 penguins, three species, four body measurements, and sex recorded for 333 of them. Download the CSV.
Requires Stratum 1.5, where every predictive model gains Validation and Test Set controls and the Model Comparison report.

The Penguin Paradox told three species apart with near-perfect accuracy. That problem is easy. Here's a harder one: from species and the four measurements, predict whether a bird is male or female. It's a two-class response with a real error floor, which is where training error starts to flatter you.

Import tip: the file writes missing values as NA. In the Import wizard, tick Treat these as missing:. Otherwise NA becomes a third value of sex, and you get a three-class problem.

Three roles, one chip

Training rows fit the model. Validation rows are held out but used to choose it: where to prune a tree, how many boosting iterations to keep. That makes their error optimistic. Test rows are never consulted. They're scored once on the finished model. Decision Tree, Boosted Trees, Random Forest and Linear Regression all carry the same Validation chip on their control bar to set this up.

1. Split once, on the Decision Tree

Analyze ▸ Data Mining ▸ Decision Tree. Set Response to sex and Predictors to species and the four measurements.
Open the Validation chip. Set Validation to 20% and Test Set to 20%. Stratify by Response is already ticked, so each set gets the same share of females and males.

Stratum Decision Tree with the Validation chip open: Validation 20%, Test Set 20%, Stratify by Response ticked — Every predictive model has the same Validation chip. The tree's header now reports training, validation and test error separately.

Stratification gives exact per-class counts: 199 training, 67 validation and 67 test rows. When there's a validation set, the tree is pruned on those rows rather than by cross-validation. Here it prunes to three leaves, with a validation error of 11.9% and a test error of 22.4%. The test number is the one to believe. The validation rows chose this tree, and 67 rows make a small jury.

2. Watch boosting overfit, then stop it

Add Analyze ▸ Data Mining ▸ Boosted Trees with the same response and predictors. Under Hyperparameters, set Trees to 300. In the Validation chip, set 20% and 20% again. Then switch View to Convergence.

Boosted Trees convergence view: training deviance falls toward zero while validation deviance bottoms out and climbs — The training curve keeps falling. The validation curve bottoms out at about 65 iterations and then climbs, which is overfitting you can see.

The training deviance keeps falling toward zero. The validation deviance reaches its minimum at about 65 iterations and then climbs, ending well above that minimum by iteration 300. Early Stopping is on by default whenever there are validation rows. It keeps only the trees up to the minimum, and the legend reads Trees 65 of 300.

Which curve to trust: with Subsample below 100%, boosting also draws a Held-out curve. That's an out-of-bag series: a row sits out one round but is fit in nearly every other. It falls along with the training curve and can't show overfitting. Only the Validation curve uses rows that never entered the fit.

3. Add a forest, then compare

Add a Random Forest with the same Validation chip settings. Then choose Analyze ▸ Data Mining ▸ Model Comparison. A new comparison includes every model card on the dataset, and the Models picker trims the list.

Model	Training error	Validation error	Test error	Test AUC
Decision Tree	12.6%	11.9%	22.4%	0.801
✓ Boosted Trees	0.0%	9.0%	10.4%	0.948
Random Forest	4.0%	9.0%	11.9%	0.946

Stratum Model Comparison report: training, validation and test error with test AUC for three models, best marked — Model Comparison reads each model's own held-out scores and recomputes nothing. The check mark goes to the lowest test error, and only when the splits match.

Training error can't rank these models: the booster scores 0.0% on rows it has already fit. On the test rows, the two ensembles are about ten points ahead of the single tree, and AUC agrees. The booster and the forest are 1.5 points apart, which is one penguin out of 67. The check mark tells you which number was lowest. It isn't a significance test.

The models held out the same rows without any extra work. Each row's role comes from its own seeded draw, not from shuffling whatever rows a model happens to use. Models with the same response and fractions therefore share a split, and changing the predictors or the filter reshuffles nobody. If two models disagree, Model Comparison adds a footnote and marks no winner.

4. Freeze the split

Click Save as Column in the chip. Stratum adds a Validation column, coded 0/1/2 and labelled Training, Validation and Test Set, and points the model at it. Choose that column as the Validation Column on any other model. From then on, changing a fraction or adding rows can't move a row between roles, and the column goes with the data if you take it into another tool.

Data grid with the saved Validation column showing Training, Validation and Test Set labels — Save as Column writes the split into the data, so any model can use it, now or later.

Linear Regression, too

In Analyze ▸ Linear Regression with Method set to Fit Model, the same chip adds a Held-out Fit section that reports n, RMSE and R² for each role. A held-out R² below zero means the model predicts those rows worse than their own mean would. Model Comparison compares models of a single response and kind, so a regression of body_mass_g gets compared with trees that also predict body_mass_g.

Why training error misleads, and how far, is the subject of the companion piece, Overfitting Explained.

Download Stratum

Frequently asked questions

What is the difference between a validation set and a test set?

Validation rows are held out of the fit but used to choose the model: how far to prune a tree, or how many boosting iterations to keep. Because they did the choosing, their error is optimistic. Test rows are never consulted. They're scored once on the finished model, which makes their error the honest out-of-sample estimate.

How large should the validation and test sets be?

Large enough that the estimate is precise enough for your decision. The standard error of an error rate p on n test rows is about √(p(1 − p)/n), so 100 rows at 12% error gives roughly ±3 points. In Stratum each held-out set can take up to 50% of the rows, and the two together up to 80%, in whole percent.

Do two models in Stratum hold out the same rows?

Yes, when they predict the same response with the same fractions. Each row's role comes from its own seeded draw, so predictors and the filter don't reshuffle anyone. Model Comparison checks this and only marks a best model when every model held out the same rows.

Does a random forest need a validation set?

Not to estimate its own error, because out-of-bag error already does that. Held-out rows matter when you compare the forest with other models: no tree bootstraps from them, and they're the same rows the other models held out.

Can I bring my own split column?

Yes. Choose it as the Validation Column. It can hold 0, 1 and 2, or the words Training, Validation and Test. A row with any other value takes no part in the fit.