Welch's ANOVA: When Variances Aren't Equal

Classic ANOVA assumes every group has the same spread. Break that assumption and it can hand you a false winner.

Follow along: channel_revenue.csv — 116 customers across three acquisition channels (Organic, Paid, Referral) with their first-month revenue. Download the CSV and run it yourself in Stratum.
Requires Stratum 1.4, where Welch's ANOVA and Games–Howell join the ANOVA controls.

You ran three acquisition channels for a month and you want to know which brings in the most valuable customers. The averages look decisive: Referral customers spend $150 in their first month against about $120 for Organic and Paid. Time to shift the budget to Referral? Run a one-way ANOVA and it seems to agree — the difference is highly significant. Except it's an illusion, and the ordinary F-test can't see why.

The catch in classical ANOVA

One-way ANOVA tests whether several group means differ, but it does so under an assumption most people never check: that every group has the same spread. It pools all the groups into a single error term, then judges every group mean against that one yardstick. When the spreads really are similar, that's efficient and fair. When they aren't, the pooled yardstick is wrong — and a small, erratic group can masquerade as a precise, reliable estimate.

That's exactly the trap in this data. Organic and Paid are large and steady; Referral is six customers whose spend runs from $64 to $243. Its average of $150 rests on a couple of big spenders, but classical ANOVA treats that shaky average as confidently as the 55-customer channels.

Check it first: Levene's test

Before trusting an ANOVA, test the assumption. In Stratum, open Analyze ▸ ANOVA ▸ One-Way, set the response to FirstMonthRevenue and the sub-groups to Channel, then tick Levene's Test (with Center = Median, the robust Brown–Forsythe variant). A section drops into the report with the verdict.

One-way ANOVA setup in Stratum with Levene's test rejecting equal variances — Levene's test (Brown–Forsythe) on the three channels: the statistic is large and p < 0.001 — equal variances rejected.

Here Levene's statistic is about 70 with a p-value below one in a million. The equal-variance assumption fails decisively. That single result is your cue: the ordinary F-test is no longer trustworthy.

The verdict flips

Look at what the two tests conclude on the very same data. Tick Welch's ANOVA and Stratum adds it as its own section, right below the classical result, so you can read them against each other:

Test	F	p-value	Conclusion
Classical ANOVA	8.25	0.0005	Significant — "the channels differ, crown Referral"
Welch's ANOVA	0.92	0.42	Not significant — the difference is within noise

Stratum report showing the classical ANOVA significant but Welch's ANOVA not significant on the same data — Same data, opposite conclusions: classical ANOVA (p = 0.0005) vs Welch's ANOVA (p = 0.42), stacked in one report.

This is the whole point. Classical ANOVA is confident the channels differ (p = 0.0005) — on that number you would move real budget to Referral. Welch's ANOVA, which weights each group by its own precision instead of pooling, says the opposite: p = 0.42, not significant. Referral's lead was a mirage produced by a tiny, high-variance sample. Using the wrong test would have handed you a costly false positive.

The post-hoc flips too: Games–Howell, not Tukey

When an omnibus test is significant you reach for a post-hoc to find which groups differ — but the standard choice, Tukey HSD, uses the same pooled error term as classical ANOVA, so it repeats the same mistake. The unequal-variance post-hoc is Games–Howell.

Tick both in Stratum and the disagreement is visible: Tukey flags Referral as significantly different from Organic and Paid — the same false winner. Games–Howell, judging each pair against its own spread, finds no reliable difference anywhere.

Games–Howell post-hoc in Stratum finding no significant pairs among the channels — Games–Howell (the unequal-variance post-hoc) finds no significant pair — while Tukey would flag Referral. Pair Games–Howell with Welch.

The rule, and how Stratum makes it one glance

The workflow is simple and worth making a habit:

Check Levene's test first. If equal variances hold, classical ANOVA and Tukey are fine.
If Levene rejects, read Welch's ANOVA instead of the classical F, and follow up with Games–Howell, not Tukey.

In Stratum these are toggles in the ANOVA controls — Levene's Test, Welch's ANOVA, Tukey, and Games–Howell each add a section to one report. You see the classical result, the assumption check, and the valid test stacked together, so the moment the verdict flips is right there on screen. No R, no SPSS, no code.

Download Stratum