Effect Size: Cohen's d, Hedges' g, and Eta Squared

A p-value says whether an effect exists. It never says how big it is. That's a different question, with different numbers.

Follow along: actg175.csv — 2,140 patients from the ACTG 175 HIV clinical trial, with baseline and week-20 CD4 counts. Download the CSV and reproduce every number below in Stratum.
Requires Stratum 1.4, where Hedges' g joins the Two-Sample report and η² / ω² / ε² join the ANOVA report.

Run enough tests on a big dataset and nearly everything comes back "significant." A p-value answers exactly one question: is this effect distinguishable from chance? At two thousand patients, the answer is almost always yes — even when the effect is trivially small. The question you actually care about is a different one: how big is it? For that you need an effect size, and it belongs next to every p-value you report.

We'll work the whole thing on one real dataset. ACTG 175 is a landmark HIV trial; the response here is the change in CD4 count from baseline to week 20 — a computed column, cd4_change = cd4_week20 − cd4_baseline, defined on the 2,139 patients who have both readings. Two questions, two effect-size families, one dataset.

Cohen's d: the standardized gap for two groups

Start with the simplest comparison: did patients on ZDV alone respond differently from patients on the other regimens? In Stratum, Analyze ▸ Two-Sample with response cd4_change and group therapy gives the raw picture. ZDV alone (n = 532) lost CD4 cells, averaging −17.07; the other regimens (n = 1,607) gained, averaging 33.34. The gap between the means is 50.4 CD4 units.

Fifty units — is that a lot? On its own the number is meaningless, because "a lot" depends on how spread out the data are. Cohen's d fixes that by dividing the gap by the pooled standard deviation, converting it into a count of standard deviations. With a pooled SD of 120.4, the 50.4-unit gap becomes:

Cohen's d = 0.42

The other regimens sit 0.42 standard deviations above ZDV alone. On Cohen's conventional scale — 0.2 small, 0.5 medium, 0.8 large — that's a moderate effect. And because d is unitless, it means the same thing whether your outcome is CD4 counts, dollars, or milliseconds. That portability is why it travels across studies and into meta-analyses.

The significance test agrees the effect is real — t = 8.37, df = 2,137, p < 0.0001 — but notice that the p-value alone would never have told you the gap is only moderate. The d does.

Hedges' g: the same idea, corrected for small samples

There's a catch with Cohen's d. On small samples it runs a little high — the pooled SD slightly underestimates the true spread, which inflates d. Hedges' g applies a correction factor, J, that shrinks d back toward the truth. It matters exactly when your data is thin.

Here it barely registers. At these sample sizes the correction factor is J = 0.9996, so Hedges' g = 0.42 — all but identical to d, a shrink of just 0.04%. That is not a flaw; it's the point. Watch what happens as the sample shrinks. The same underlying d of 0.42 would come out as g ≈ 0.41 at n = 40 (J = 0.980), 0.40 at n = 20 (J = 0.958), and 0.39 at n = 12 (J = 0.923). Invisible at two thousand patients; real at a dozen. Reach for g by default and you're covered either way.

Eta squared: effect size for three or more groups

Cohen's d compares two groups. When you have three or more, the natural effect size is eta squared (η²) — the share of the outcome's total variance that the grouping explains. ACTG 175 actually ran four treatment arms, so switch to Analyze ▸ ANOVA ▸ One-Way with response cd4_change and factor arm (ZDV + ddC, ddI only, ZDV only, and ZDV + ddI).

Stratum One-Way ANOVA report with the eta squared column reading 0.043 alongside F = 31.98 and p near zero — The One-Way ANOVA report: F = 31.98 with p ≈ 0, yet the η² column reads just 0.043 — the arm explains about 4.3% of the variance.

The omnibus test is emphatic: F = 31.98, df = (3, 2,135), p ≈ 0. By the p-value alone you'd declare the arms wildly different. Now read the η² column: η² = 0.043. The treatment arm explains only about 4.3% of the variance in CD4 change — small on the conventional 0.01 / 0.06 / 0.14 scale. Stratum reports two less-biased cousins beside it, ω² = 0.042 and ε² = 0.042; at this sample size all three nearly coincide. A near-zero p-value and a small effect, side by side, on the same test.

How small is small? For contrast, flipper length across penguin species yields η² ≈ 0.78 — roughly 78% of the variance explained, an enormous effect. Same statistic, opposite end of the scale. That's the range η² is built to describe, and why 0.043 is a genuinely modest result no matter how tiny the p-value gets.

Report the effect size next to every p-value

The two questions are separate, and you need both answers. Whether an effect exists is the p-value's job; how big it is is the effect size's. On a large dataset the first answer is nearly always "yes," which is precisely why it's only half the story — the arm ANOVA is the honest warning: F = 32, p ≈ 0, yet η² = 0.043. Report the p-value alone and a 4%-of-variance effect reads as a triumph.

The habit is simple and worth making permanent:

Two groups → Cohen's d, or Hedges' g when the sample is small (and g is a safe default at any size).
Three or more groups → eta squared (or the less-biased ω² / ε²).
Always → print the effect size right beside the p-value, so magnitude and significance are read together.

In Stratum these aren't a separate calculation. The Two-Sample report carries Cohen's d and Hedges' g next to the t-test; the One-Way ANOVA report carries η², ω², and ε² next to F. Whether an effect exists and how big it is arrive in the same report — no code, no second tool.

Download Stratum