Validation Record

What every statistic in Stratum is checked against — and where Stratum deliberately differs from other packages.

Every analysis is verified against values computed outside Stratum: by R, SciPy, NumPy, scikit-learn, or the certified NIST Statistical Reference Datasets. Those reference values are committed to the source tree and asserted on every build, so a result cannot quietly stop matching them.

We do not claim that Stratum “matches R”. That claim is not well defined. R's own functions disagree with each other in places — quantile() and boxplot() use different quartiles — and the same rule names mean different things in different packages. What is published here instead is the specific claim, analysis by analysis, naming the tool and the function, together with every place we chose differently and why.

48 analyses. 19 checked against an outside tool · 12 partly checked · 3 pinned by a published formula · 14 no statistic of its own. This page is generated from the same ledger the test suite emits, so it cannot describe evidence that is not there. Last generated 2026-08-31. Reference tool versions: ca (R package) 0.71.1, mpmath 1.4.1, numpy 2.4.6, pingouin 0.6.1, R 4.6.1, scikit-learn 1.9.0, scipy 1.16.3, statsmodels 0.14.6.

How to read this

Checked against an outside tool (19)
Committed reference values from a named function of a named package at a recorded version, asserted on every build.
Partly checked (12)
Some published outputs are verified and others are not — and the entry says which. A fitted random forest cannot be matched tree-for-tree against another library, so its diagnostics are checked and the model itself rests on invariants.
Pinned by a published formula (3)
No package computes the quantity, so the published formula is transcribed independently twice and the two must agree.
No statistic of its own (14)
The chart draws values it was handed. A scatter plot has nothing to verify beyond the numbers given to it.

“Partly checked” is not a disclaimer added after the fact — it names work that is still outstanding, deliberately, because a verification claim later found to be wrong costs more than one never made.

Deliberate divergences

If you reconcile a Stratum result against another package and the numbers differ slightly, these are the likely reasons. Each is a documented choice.

Analysis by analysis

Analysis Status Checked against
SummaryVerified
  • scipy.stats.skew(bias=False)
  • scipy.stats.kurtosis(bias=False, fisher=True)
HistogramVerified
  • R: ceiling(log2(n)+1), 3.49*sd*n^(-1/3), 2*IQR*n^(-1/3) from its own sd/IQR; nclass.Sturges / nclass.scott / nclass.FD recorded beside them (nclass.scott uses 3.5, not 3.49)
  • numpy: the same three rules spelled independently
  • the KDE overlay routes through the shared KernelDensityEngine
Heatmap/HexBinVerified
  • bins BOTH axes with the Freedman-Diaconis rule, checked against R nclass.FD and numpy
MosaicVerified
  • R stats::chisq.test(N, correct=FALSE) — $expected and $residuals, the PEARSON residuals (O-E)/sqrt(E), which is what Stratum reports; $stdres is a different quantity and is recorded alongside for contrast
  • scipy.stats.chi2_contingency for the expected counts, with the residuals formed from them independently
Box-WhiskerVerified
  • R stats::quantile(type=7) — NOT fivenum/boxplot, which use Tukey's hinges and give different quartiles at some sample sizes
  • numpy.quantile (linear == type 7)
  • scipy.stats.skew / kurtosis (bias=False)
  • R moments::skewness / kurtosis, converted from the biased g1/b2 to the unbiased G1/G2
  • McGill-Tukey-Larsen (1978) for the notch, pinned by formula
ViolinVerified
  • the density envelope uses the same kernel-density estimator as the Density chart; the inner box statistics are Box-Whisker's, covered there
DensityVerified
  • R: 0.9*min(sd, IQR/1.3489795)*n^-0.2 and (4/(3n))^0.2*sd from its own sd/IQR, plus the exact kernel sum mean(dnorm((t-x)/h))/h
  • numpy: the same rules and kernel sum spelled independently
  • R bw.nrd0 / bw.nrd recorded alongside — they divide by 1.34, not the exact 1.3489795, so they sit 0.67% away once the IQR branch wins
ECDFVerified
  • R stats::ecdf(x) evaluated at the distinct values and outside both ends
  • numpy (x <= t).sum()/n
CorrelationVerified
  • R stats::cor.test (pearson / spearman / kendall)
  • scipy.stats.pearsonr / spearmanr / kendalltau
  • Bonett-Wright (2000) for the Spearman interval
One-Way ANOVAVerified
  • scipy.stats.f_oneway
  • scipy.stats.kruskal
  • eta^2 / omega^2 / epsilon^2 in exact Fraction arithmetic
  • pingouin.welch_anova (cross-checked against the definition)
  • pingouin.pairwise_gameshowell
  • scipy.stats.studentized_range.sf / .ppf
Two-Way ANOVAVerified
  • statsmodels.stats.anova_lm(typ=3) on a sum-coded model (== R car::Anova)
  • pooled within-groups SS in exact rational arithmetic
MANOVAVerified
  • statsmodels.multivariate.manova.MANOVA.mv_test()
  • 50-digit mpmath for Rao's F degrees of freedom
Two-SampleVerified
  • scipy.stats.mannwhitneyu
  • scipy.stats.wilcoxon
  • scipy.stats.ttest_ind
  • scipy.stats.norm.sf
DifferencesVerified
  • R stats::chisq.test(correct=FALSE) with V = sqrt(chi2/(n*(k-1)))
  • scipy.stats.chi2_contingency and scipy.stats.contingency.association(method='cramer')
Contingency TableVerified
  • scipy.stats.chi2_contingency(correction=False)
  • scipy.stats.chi2.sf
Principal ComponentsVerified
  • R stats::prcomp(center=TRUE, scale.=TRUE/FALSE)
  • sklearn.decomposition.PCA
  • both re-anchored to Stratum's sign convention, so loadings and scores compare element by element rather than as magnitudes
Linear RegressionVerified
  • NIST StRD Linear Least Squares — certified to 15 digits (Norris, NoInt1, Filip, Longley, Wampler1-5)
Process CapabilityVerified
  • c4 via scipy.special.gammaln
  • d2 by Gauss-Legendre quadrature of the range distribution
  • numpy sample SD (ddof=1)
Regression PlotVerified
  • R lm(): fitted(), resid(), hatvalues(), rstandard() (internally studentized), rstudent() (externally studentized), cooks.distance() — the plots arrange these and compute nothing of their own
Q-QPartly verified
  • R: (i-0.5)/n plotting positions, qnorm / -log(1-p) / p / log(-log(1-p)) for the theoretical axes
  • R ppoints() recorded alongside — it uses a = 3/8 at n <= 10, so it agrees with Stratum only above that
  • normalQuantile pinned to Acklam's published 1.15e-9 RELATIVE bound against R qnorm

Not yet covered: The CONFIDENCE BAND (QQConfidenceBand) has no fixture — only the points and the theoretical axes are covered.

TrendPartly verified
  • R stats::lm for all six parametric forms — raw-power polynomials, lm(y ~ log(x)), lm(log(y) ~ log(x)), lm(log(y) ~ x)
  • r-squared recomputed on the ORIGINAL y scale against the back-transformed prediction, which is what Stratum reports; R's model-scale summary()$r.squared is recorded beside it and asserted to differ for exactly the power and exponential forms

Not yet covered: The non-parametric smoothers — LOWESS, kernel smoother and moving average — have no fixture. Their implementations differ in detail between packages, so pinning one spelling of LOWESS would pin R's choices rather than a shared definition; closing this means deciding which definition Stratum claims first.

Contour PlotPartly verified
  • grid geometry from the engine's own 2-D rule, h = 1.06*max(sd, range*1e-3)*n^(-1/6)*scale, pad = clamp(2.5h, 3%, 15%) — the bandwidth is not exposed, but x0 and dx are functions of it, so the geometry pins the rule exactly
  • a DIRECT 2-D Gaussian kernel sum in numpy, compared in shape against the engine's binned separable blur within a measured bound (6%, 12% on the coarse grid)

Not yet covered: The blur is an APPROXIMATION to a 2-D KDE, so the surface is checked for shape against the exact estimator rather than value-for-value — the bound is a recorded measurement, not an equality. The value and aggregate modes (with a Z column) and their 2% masking threshold have no fixture. Note this chart does NOT share KernelDensityEngine: it uses the 2-D n^(-1/6) exponent where Density uses n^(-1/5), so it inherits nothing from that fixture.

Decision TreePartly verified
  • sklearn.metrics.roc_curve(drop_intermediate=False) / roc_auc_score
  • sklearn.inspection.partial_dependence(kind='average')

Not yet covered: The DIAGNOSTICS (ROC, AUC, calibration binning, partial dependence) are pinned; the fitted TREE is not. A single tree can be matched to sklearn exactly given the same split rule and deterministic tie-breaking, so this is a gap to close, not an impossibility.

Random ForestPartly verified
  • sklearn.metrics.roc_curve / roc_auc_score

Not yet covered: Diagnostics only. An ensemble cannot be matched tree-for-tree against sklearn — the RNG streams and split tie-breaking differ by construction — so the model itself needs invariants (OOB convergence, importance monotonicity) rather than a value comparison.

Boosted TreesPartly verified
  • sklearn.metrics.roc_curve / roc_auc_score

Not yet covered: Diagnostics only — same reasoning as Random Forest.

Correspondence AnalysisPartly verified
  • R ca::ca()
  • an independent numpy implementation from the definition (Greenacre), not a transcription of the engine
  • scipy.stats.chi2_contingency — pins the identity n * total inertia == chi-square

Not yet covered: The DISPLAY CAP is not covered. Above StratumLimits.maxReportGroups the engine pools the tail into 'Other' and decomposes the POOLED grid, which moves both the inertia and the degrees of freedom; every fixture table is well under the cap. Row/column contributions and quality (cos²) have no fixture yet either.

ClusteringPartly verified
  • R stats::hclust(dist(X), method=single/complete/average/ward.D2)
  • scipy.cluster.hierarchy.linkage
  • sklearn.cluster.KMeans (well-separated data only)
  • local optimality: nearest-centroid assignment, and centroid is the cost-minimising centre — mean under Euclidean, median under Manhattan

Not yet covered: The k-means PARTITION is compared to scikit-learn only where the optimum is unambiguous. On ambiguous data, and under the Manhattan cost (which sklearn's KMeans does not implement), the evidence is invariants rather than an outside implementation — because no outside implementation of that estimator exists to compare against, not because the check was skipped. Elbow and silhouette have no fixture yet.

I-MR ChartPartly verified
  • d2 by Gauss-Legendre quadrature
  • c4 via scipy.special.gammaln

Not yet covered: The unbiasing CONSTANTS are pinned to first principles; the Nelson / Western Electric run rules are not, and they are a documented convention rather than a computed quantity.

X-bar ChartPartly verified
  • d2 / c4 as for I-MR

Not yet covered: Constants pinned; run rules are convention. Same as I-MR.

P ChartPartly verified
  • binomial limits

Not yet covered: Attribute-chart limits are not yet cross-checked against an outside tool.

C ChartPartly verified
  • Poisson limits

Not yet covered: Attribute-chart limits are not yet cross-checked against an outside tool.

ParetoBy formula

ranking and a running total are exact arithmetic, so there is no outside tool to disagree with. What is pinned is falsifiable and was: the tie-break is the CATEGORY LABEL (so equal frequencies order deterministically rather than by dictionary order), and topN truncation does NOT renormalise — the cumulative line runs against the FULL total and stops short of 100%. Renormalising it trips 42 assertions.

CUSUM ChartBy formula

Montgomery's CUSUM recursion — C+ = max(0, C+ + z - k), C- = max(0, C- - z - k), FIR headstart h/2 — transcribed INDEPENDENTLY in R and numpy and asserted equal before either was recorded. No qcc/spc is installed, so this is the published formula rather than a package's implementation; installing either upgrades this to external. The sigma it standardises by is MRbar/d2, which shared_estimators pins to first principles. NOT covered: the ARL.

EWMA ChartBy formula

Montgomery's EWMA recursion with the TIME-VARYING limit width L*sigma*sqrt(lambda/(2-lambda)*(1-(1-lambda)^(2i))), transcribed independently in R and numpy. The test additionally asserts the band widens monotonically, so substituting the asymptotic width fails even where the two happen to be close. Same qcc/spc caveat as CUSUM.

DataNo statistic

The data sheet. Displays stored values.

Dot/StripNo statistic

Plots raw values along one axis.

PieNo statistic

Shares of a supplied total.

TreemapNo statistic

Areas proportional to a supplied value.

BarNo statistic

Plots the aggregate the control bar selects; the aggregates themselves are Summary's, and covered there.

Dot PlotNo statistic

Plots raw values along one axis.

DumbbellNo statistic

Draws two supplied values per category.

SlopegraphNo statistic

Draws two supplied values per category.

ScatterNo statistic

Plots raw x/y pairs. Any fitted overlay is a separate mode with its own entry.

BubbleNo statistic

Plots raw x/y pairs with a size channel.

PCA Scree PlotNo statistic

Plots the PCA eigenvalues, which are covered under Principal Components.

PCA BiplotNo statistic

Plots the PCA loadings and scores, which are covered under Principal Components.

Correspondence MapNo statistic

Plots the correspondence-analysis coordinates, which are covered under Correspondence Analysis.

Parallel PlotNo statistic

Draws one polyline per row over scaled axes.

Finding a disagreement

If a Stratum result differs from another package in a way this page does not explain, please tell us — send the data and both results to support.