StratumFisher's Exact Test vs Chi-Square: When Small Counts Break χ²
Same 2×2 table, two p-values on opposite sides of 0.05. The expected counts tell you which one to believe.
Requires Stratum 1.5, where the Contingency Table gains its Statistics and Residuals options.
Are fans of the Star Wars Expanded Universe also Star Trek fans? Across the whole survey the answer is a modest, borderline maybe. Narrow it to one census region, West South Central, and a Pearson chi-square test says yes: p = 0.038. Fisher's exact test, on the very same table, says p = 0.063. One clears the conventional bar and one doesn't. They can't both be the number you report.
When the approximation holds, the tests agree
Start with the full survey. In Stratum, open Analyze ▸ Contingency Table and set Sub-Group (Rows) to the Expanded Universe fan question and Parameter (Columns) to the Star Trek fan question (renamed EU fan and Trek fan here). Then open the Show options and tick Expected and Statistics.
213 people answered both questions. Among Expanded Universe fans 72.7% are also Trek fans, against 60.5% of the rest. The smallest expected count is 33.5, and the tests line up: Pearson χ² p = 0.060, Yates p = 0.083, Fisher p = 0.081. Nothing to adjudicate, because Pearson's statistic really is close to χ²-distributed at these counts.
Shrink the sample and Cochran's rule breaks
Now ask the regional question. In Organize, switch the filter to Keep and add a rule on Region that is one of West South Central. Twenty-one respondents remain, and the expected counts collapse to 2.86, 7.14, 3.14 and 7.86. Half the cells expect fewer than five.
Cochran's guideline asks that no more than a fifth of cells expect fewer than five (and none fewer than one). It exists because Pearson's X² is a sum over a discrete table that is only asymptotically χ²: with expectations this small its distribution is lumpy, and the continuous curve misstates the tail. Stratum applies the same 20% threshold. Tick Sparse Cell % and the stats line reads 50.0%, and the report footnotes that more than 20% of cells expect fewer than five, where Cramér's V is biased upward (it also gives the Bergsma-corrected V).
Four p-values, one table
With Statistics on, the Tests section prints four rows:
| Test | Statistic | p-value | What it is |
|---|---|---|---|
| Pearson χ² | 4.30 | 0.038 | Asymptotic; too eager here |
| Likelihood ratio G² | 4.56 | 0.033 | Asymptotic; no better at small N |
| Yates continuity correction | 2.52 | 0.112 | Corrected approximation; conservative |
| Fisher's exact test | — | 0.063 | Exact, conditional on the margins |
Fisher's exact test conditions on both margins, which makes the top-left count hypergeometric. Here it can only be 0 through 6, and the observed 5 has probability 0.051. Nothing is approximated, so there is no expected-count rule to violate. Fisher is itself somewhat conservative, since conditioning leaves a coarse discrete distribution; unconditional exact tests such as Boschloo's give about 0.048 on this table, but Stratum's report doesn't include them.
Yates's correction shaves half a count off each |O − E| to mimic that discreteness, and overshoots: 0.112 is further from the exact answer than Pearson is. Stratum caps the correction at |O − E| as R's chisq.test does, so a table already within half a count of independence reads χ² = 0 rather than being corrected past it.
fisher.test rule, and the report's footnote states it. Doubling the one-sided p instead gives 0.110 on this asymmetric table. When a reviewer's number doesn't match yours, check this first.Effect size: odds ratio and relative risk
A p-value straddling 0.05 is the least interesting fact about this table. On any 2×2, Statistics also adds a Risk Estimate section. Rows are the exposure (EU fan) and columns the outcome (Trek fan), and each label spells out which way it's divided:
- Odds ratio (No / Yes): 10, 95% CI 0.91 to 110.
- Relative risk of Yes (No / Yes): 0.55, CI 0.29 to 1.05. Turned the other way up, Expanded Universe fans are 1.82 times as likely to be Trek fans (CI 0.95 to 3.47).
Both intervals cross 1, consistent with Fisher rather than with Pearson. The odds ratio is the sample ratio ad/bc with a Woolf (log-scale Wald) interval; the relative risks use the Katz log interval. That is the convention SPSS and epitools follow. R's fisher.test instead prints the conditional maximum-likelihood odds ratio, 8.89, with an exact interval of 0.72 to 517. It's a different estimator, not a disagreement. If a cell is zero, Stratum leaves the estimate blank and adds a footnote rather than reporting 0 or infinity.
Which cells drive it: Residuals
Tick Residuals and each cell gets a second line: (O − E) ÷ √E. Here they run from −1.21 to 1.27, and none reaches the ±2 the footnote suggests as a guide. These are Pearson residuals (SPSS's “Standardized”), not the adjusted residuals R reports as $stdres; on a 2×2 the adjusted residual is ±√X², or 2.07 here. Know which one you're quoting.
The rule
- Look at the expected counts before the p-value. Tick Expected or Sparse Cell %.
- Healthy counts: Pearson's χ² is fine, and on a 2×2 Fisher will agree with it anyway.
- More than a fifth of cells under five: report Fisher's exact test, not Pearson, and not Yates as a compromise.
- Always report the odds ratio or relative risk with its interval. At small N the interval is the finding.
- Subgroups you picked after looking are their own multiple-comparisons problem. This one illustrates the test; it is not a finding about the region.
Stratum doesn't choose for you: one report prints the cross-tab, all four tests, phi, the contingency coefficient, Cramér's V and the risk estimates. The same Contingency Table, with the same Show options, runs on Mac and on Windows, and its 2×2 statistics are checked against R in Stratum's validation record.
Frequently asked questions
When should I use Fisher's exact test instead of chi-square?
When the expected counts are small. The usual guide is Cochran's: chi-square is trustworthy when no more than 20% of cells expect fewer than five and none expects fewer than one. On a 2×2 that means any cell expecting fewer than five is a reason to report Fisher's exact test.
Should I use Yates's continuity correction?
Rarely as your headline test. Yates's correction pulls the chi-square approximation toward the exact answer but overshoots, so it is conservative. If you are worried enough about small counts to correct, Fisher's exact test answers the question directly.
Why do different programs give different two-sided Fisher p-values?
Because two-sided has more than one definition. Stratum, like R's fisher.test, sums the probability of every table with the same margins that is no more likely than the observed one. Some tools double the one-sided p-value instead, which gives a larger number on asymmetric tables (0.110 versus 0.063 here).
Why doesn't Stratum's odds ratio match R's fisher.test?
They are different estimators. Stratum reports the sample odds ratio ad/bc with a Woolf 95% interval, as SPSS and epitools do. R's fisher.test reports the conditional maximum-likelihood odds ratio with an exact interval: 10 [0.91, 110] versus 8.89 [0.72, 517] on this table.
What are the residuals in Stratum's Contingency Table?
Pearson residuals, (observed − expected) ÷ √expected — SPSS's Standardized Residual and R's chisq.test()$residuals. They are not the adjusted residuals R reports as $stdres, which are larger.
Related: The Star Wars Map: A Whole Survey in One Chart → · What Is Correspondence Analysis? → · Effect Size: Cohen's d, Hedges' g, and Eta Squared → · Which Statistical Test? →