Random Forest & Boosted Trees: Stronger Predictions

In the last lesson we built a single decision tree and admired how readable it was. Its weakness is the flip side of that simplicity: a single tree is unstable. Swap out a handful of rows and the top split can change, cascading into a completely different tree. Ensembles cure this by training many trees and pooling their answers, so no single quirky tree gets the final say. Stratum offers the two classic approaches — the random forest and boosted trees — both written in pure Swift, no Python required. We'll use the UCI Wine data and predict which of three cultivars each wine came from.

Bagging: the random forest

Open it from Analyze ▸ Data Mining ▸ Random Forest. A random forest builds its trees in parallel, and it deliberately makes each one a little different. Every tree is trained on a random bootstrap sample — rows drawn at random, with replacement — and at each split it's only allowed to consider a random subset of the variables (the Features control; Trees sets how many trees to grow). This double dose of randomness is called bagging, and it's the whole trick: because the trees disagree in different ways, their individual errors tend to cancel out when you average their votes. The forest is far steadier than any one tree inside it.

Out-of-bag error: validation for free

Bagging hands you a bonus. Because each tree skips roughly a third of the rows when it's trained, every row has a set of trees that never saw it. Stratum quietly tests each row against exactly those trees and reports the out-of-bag error — an honest accuracy estimate with no separate validation split needed. Both ensembles open on the Importance view; switch View ▸ Convergence to plot the OOB Error Rate, and watch it fall as more trees join the forest and flatten out once adding trees stops helping.

Boosting: trees that learn from mistakes

Boosted trees (Analyze ▸ Data Mining ▸ Boosted Trees) take the opposite tack. Instead of building independent trees in parallel, boosting grows them sequentially, and each new tree is trained to fix the errors the ensemble has made so far. The trees are usually tiny, and they're added one at a time, each nudging the predictions a little closer to the truth. This gradient boosting often squeezes out higher accuracy than a forest — at the cost of more careful tuning, since too many trees can eventually start fitting the noise.

The convergence curve tells the story — it's the same View ▸ Convergence plot (OOB Error Rate on the y-axis) you used for the forest. As trees are added, error drops steeply at first, then levels off. The sweet spot is where the curve flattens — enough trees to be accurate, not so many that the model starts memorizing. Stratum plots it for you so the right stopping point is obvious.

Importance without the picture

You give up the single readable flowchart when you move to an ensemble — there's no one tree to trace anymore. What you keep is feature importance, which is the default view both ensembles open on (View ▸ Importance). Stratum averages each variable's impurity reductions across every tree in the ensemble, giving a far more stable ranking than a lone tree could. For the wine data, a couple of chemical measurements dominate, and the bar chart says so at a glance.

Forest or boosting — which one?

As a rule of thumb: reach for a random forest first. It's robust, hard to overfit, needs little tuning, and gives you out-of-bag error for free. Move to boosted trees when you want to wring out the last bit of accuracy and you're willing to watch the convergence curve and tune. And remember the trade-off both share: more accuracy, less interpretability. When you simply need to explain a single decision, the lone tree from the last lesson still wins.

We've now climbed the whole modeling ladder, from a single transparent tree to powerful ensembles. In the final lesson we change gears entirely and ask a different question: can Stratum stay fast and fluid when the dataset has five million rows?