Understand the failure.
Verify the repair.
Small, reproducible software failures. The broken implementation, the fix that didn’t work, and the one that passed—preserved together.
Explore the cases ↓How results are verified ↗WHAT THE ARCHIVE CONTAINS
100840 executable cases. 20168 are open.
Every case records the implementation that fails, the fix that did not work, and the repair that passed its checks—with recorded outputs and source hashes. This release adds 100840 cases across 20168 failure mechanisms and 254 domains.
The open tier gives you the failure and the unsuccessful fix for one case in every mechanism. The remaining 80672 cases, 5 variants per mechanism, are member-only: the verified repair, its recorded checks, and the full fixture suite are held in the member archive. Read the methodology ↗
A RECORD OF WHAT WENT WRONG
Browse the archive / 100840
Welch unequal-variance t statistic: The t denominator mixes variances before dividing · case 01
Unequal arm sizes give a standard error that is neither pooled nor Welch.
Welch unequal-variance t statistic: The t denominator mixes variances before dividing · case 02
Unequal arm sizes give a standard error that is neither pooled nor Welch.
Welch unequal-variance t statistic: The t denominator mixes variances before dividing · case 03
Unequal arm sizes give a standard error that is neither pooled nor Welch.
Welch unequal-variance t statistic: The t denominator mixes variances before dividing · case 04
Unequal arm sizes give a standard error that is neither pooled nor Welch.
Welch unequal-variance t statistic: The t denominator mixes variances before dividing · case 05
Unequal arm sizes give a standard error that is neither pooled nor Welch.
Welch unequal-variance t statistic: Welch df numerator uses raw variances · case 01
Degrees of freedom scale with the metric's magnitude and sample size in nonsensical ways.
Welch unequal-variance t statistic: Welch df numerator uses raw variances · case 02
Degrees of freedom scale with the metric's magnitude and sample size in nonsensical ways.
Welch unequal-variance t statistic: Welch df numerator uses raw variances · case 03
Degrees of freedom scale with the metric's magnitude and sample size in nonsensical ways.
Welch unequal-variance t statistic: Welch df numerator uses raw variances · case 04
Degrees of freedom scale with the metric's magnitude and sample size in nonsensical ways.
Welch unequal-variance t statistic: Welch df numerator uses raw variances · case 05
Degrees of freedom scale with the metric's magnitude and sample size in nonsensical ways.
Relative lift interval: Control noise is scaled like treatment noise · case 01
Intervals are too narrow whenever the lift is large.
Relative lift interval: Control noise is scaled like treatment noise · case 02
Intervals are too narrow whenever the lift is large.
Relative lift interval: Control noise is scaled like treatment noise · case 03
Intervals are too narrow whenever the lift is large.
Relative lift interval: Control noise is scaled like treatment noise · case 04
Intervals are too narrow whenever the lift is large.
Relative lift interval: Control noise is scaled like treatment noise · case 05
Intervals are too narrow whenever the lift is large.
Relative lift interval: Unit variances are used as variances of the mean · case 01
Lift intervals do not shrink as more users are enrolled.
Relative lift interval: Unit variances are used as variances of the mean · case 02
Lift intervals do not shrink as more users are enrolled.
Relative lift interval: Unit variances are used as variances of the mean · case 03
Lift intervals do not shrink as more users are enrolled.
Relative lift interval: Unit variances are used as variances of the mean · case 04
Lift intervals do not shrink as more users are enrolled.
Relative lift interval: Unit variances are used as variances of the mean · case 05
Lift intervals do not shrink as more users are enrolled.
Relative lift interval: Lift is reported as an absolute difference · case 01
A relative-lift dashboard shows raw metric deltas as if they were percentages.
Relative lift interval: Lift is reported as an absolute difference · case 02
A relative-lift dashboard shows raw metric deltas as if they were percentages.
Relative lift interval: Lift is reported as an absolute difference · case 03
A relative-lift dashboard shows raw metric deltas as if they were percentages.
Relative lift interval: Lift is reported as an absolute difference · case 04
A relative-lift dashboard shows raw metric deltas as if they were percentages.
Relative lift interval: Lift is reported as an absolute difference · case 05
A relative-lift dashboard shows raw metric deltas as if they were percentages.
Relative lift interval: The half-width multiplies z by the variance · case 01
Intervals collapse for small variances and explode for large ones.
Relative lift interval: The half-width multiplies z by the variance · case 02
Intervals collapse for small variances and explode for large ones.
Relative lift interval: The half-width multiplies z by the variance · case 03
Intervals collapse for small variances and explode for large ones.
Relative lift interval: The half-width multiplies z by the variance · case 04
Intervals collapse for small variances and explode for large ones.
Relative lift interval: The half-width multiplies z by the variance · case 05
Intervals collapse for small variances and explode for large ones.
CUPED covariate adjustment: Theta is normalised by the outcome variance · case 01
The adjustment coefficient is wrong whenever outcome and covariate scales differ.
CUPED covariate adjustment: Theta is normalised by the outcome variance · case 02
The adjustment coefficient is wrong whenever outcome and covariate scales differ.
CUPED covariate adjustment: Theta is normalised by the outcome variance · case 03
The adjustment coefficient is wrong whenever outcome and covariate scales differ.
CUPED covariate adjustment: Theta is normalised by the outcome variance · case 04
The adjustment coefficient is wrong whenever outcome and covariate scales differ.
CUPED covariate adjustment: Theta is normalised by the outcome variance · case 05
The adjustment coefficient is wrong whenever outcome and covariate scales differ.
CUPED covariate adjustment: Covariates are centred on their own arm mean · case 01
The adjustment cancels out and CUPED leaves the estimate unchanged.
CUPED covariate adjustment: Covariates are centred on their own arm mean · case 02
The adjustment cancels out and CUPED leaves the estimate unchanged.
CUPED covariate adjustment: Covariates are centred on their own arm mean · case 03
The adjustment cancels out and CUPED leaves the estimate unchanged.
CUPED covariate adjustment: Covariates are centred on their own arm mean · case 04
The adjustment cancels out and CUPED leaves the estimate unchanged.
CUPED covariate adjustment: Covariates are centred on their own arm mean · case 05
The adjustment cancels out and CUPED leaves the estimate unchanged.
CUPED covariate adjustment: Theta is fitted on the control arm only · case 01
The adjustment ignores treatment data and centres on the control covariate mean.
CUPED covariate adjustment: Theta is fitted on the control arm only · case 02
The adjustment ignores treatment data and centres on the control covariate mean.
CUPED covariate adjustment: Theta is fitted on the control arm only · case 03
The adjustment ignores treatment data and centres on the control covariate mean.
CUPED covariate adjustment: Theta is fitted on the control arm only · case 04
The adjustment ignores treatment data and centres on the control covariate mean.
CUPED covariate adjustment: Theta is fitted on the control arm only · case 05
The adjustment ignores treatment data and centres on the control covariate mean.
CUPED covariate adjustment: A constant covariate gets a unit coefficient · case 01
Experiments whose covariate never varies report theta = 1.
CUPED covariate adjustment: A constant covariate gets a unit coefficient · case 02
Experiments whose covariate never varies report theta = 1.
CUPED covariate adjustment: A constant covariate gets a unit coefficient · case 03
Experiments whose covariate never varies report theta = 1.
CUPED covariate adjustment: A constant covariate gets a unit coefficient · case 04
Experiments whose covariate never varies report theta = 1.
CUPED covariate adjustment: A constant covariate gets a unit coefficient · case 05
Experiments whose covariate never varies report theta = 1.
Group sequential stopping rule: A statistic equal to the boundary does not stop · case 01
The trial runs past a look whose statistic lands exactly on the boundary.
Group sequential stopping rule: A statistic equal to the boundary does not stop · case 02
The trial runs past a look whose statistic lands exactly on the boundary.
Group sequential stopping rule: A statistic equal to the boundary does not stop · case 03
The trial runs past a look whose statistic lands exactly on the boundary.
Group sequential stopping rule: A statistic equal to the boundary does not stop · case 04
The trial runs past a look whose statistic lands exactly on the boundary.
Group sequential stopping rule: A statistic equal to the boundary does not stop · case 05
The trial runs past a look whose statistic lands exactly on the boundary.
Group sequential stopping rule: Futility can fire at the final look · case 01
Final analyses report futility instead of the null outcome.
Group sequential stopping rule: Futility can fire at the final look · case 02
Final analyses report futility instead of the null outcome.
Group sequential stopping rule: Futility can fire at the final look · case 03
Final analyses report futility instead of the null outcome.
Group sequential stopping rule: Futility can fire at the final look · case 04
Final analyses report futility instead of the null outcome.
Group sequential stopping rule: Futility can fire at the final look · case 05
Final analyses report futility instead of the null outcome.
Group sequential stopping rule: Every look uses the final boundary · case 01
Early looks stop at the much lower final threshold, inflating type I error.
Group sequential stopping rule: Every look uses the final boundary · case 02
Early looks stop at the much lower final threshold, inflating type I error.
Group sequential stopping rule: Every look uses the final boundary · case 03
Early looks stop at the much lower final threshold, inflating type I error.
Group sequential stopping rule: Every look uses the final boundary · case 04
Early looks stop at the much lower final threshold, inflating type I error.
Group sequential stopping rule: Every look uses the final boundary · case 05
Early looks stop at the much lower final threshold, inflating type I error.
Group sequential stopping rule: Continuing trials report the next look number · case 01
Monitoring shows one more completed look than actually happened.
Group sequential stopping rule: Continuing trials report the next look number · case 02
Monitoring shows one more completed look than actually happened.
Group sequential stopping rule: Continuing trials report the next look number · case 03
Monitoring shows one more completed look than actually happened.
Group sequential stopping rule: Continuing trials report the next look number · case 04
Monitoring shows one more completed look than actually happened.
Group sequential stopping rule: Continuing trials report the next look number · case 05
Monitoring shows one more completed look than actually happened.
Group sequential stopping rule: The final look without efficacy keeps the trial running · case 01
Experiments never conclude after the last planned analysis.
Group sequential stopping rule: The final look without efficacy keeps the trial running · case 02
Experiments never conclude after the last planned analysis.
Group sequential stopping rule: The final look without efficacy keeps the trial running · case 03
Experiments never conclude after the last planned analysis.
Group sequential stopping rule: The final look without efficacy keeps the trial running · case 04
Experiments never conclude after the last planned analysis.
Group sequential stopping rule: The final look without efficacy keeps the trial running · case 05
Experiments never conclude after the last planned analysis.
Holm correction across metrics: Rejection continues after a failed step · case 01
A large p-value followed by a lucky threshold rejects metrics Holm would keep.
Holm correction across metrics: Rejection continues after a failed step · case 02
A large p-value followed by a lucky threshold rejects metrics Holm would keep.
Holm correction across metrics: Rejection continues after a failed step · case 03
A large p-value followed by a lucky threshold rejects metrics Holm would keep.
Holm correction across metrics: Rejection continues after a failed step · case 04
A large p-value followed by a lucky threshold rejects metrics Holm would keep.
Holm correction across metrics: Rejection continues after a failed step · case 05
A large p-value followed by a lucky threshold rejects metrics Holm would keep.
Holm correction across metrics: Every step uses the Bonferroni threshold · case 01
Metrics that Holm would reject after the first step stay unrejected.
Holm correction across metrics: Every step uses the Bonferroni threshold · case 02
Metrics that Holm would reject after the first step stay unrejected.
Holm correction across metrics: Every step uses the Bonferroni threshold · case 03
Metrics that Holm would reject after the first step stay unrejected.
Holm correction across metrics: Every step uses the Bonferroni threshold · case 04
Metrics that Holm would reject after the first step stay unrejected.
Holm correction across metrics: Every step uses the Bonferroni threshold · case 05
Metrics that Holm would reject after the first step stay unrejected.
Holm correction across metrics: Adjusted p-values are not made monotone · case 01
A metric with a smaller raw p-value reports a larger adjusted p-value than a later one.
Holm correction across metrics: Adjusted p-values are not made monotone · case 02
A metric with a smaller raw p-value reports a larger adjusted p-value than a later one.
Holm correction across metrics: Adjusted p-values are not made monotone · case 03
A metric with a smaller raw p-value reports a larger adjusted p-value than a later one.
Holm correction across metrics: Adjusted p-values are not made monotone · case 04
A metric with a smaller raw p-value reports a larger adjusted p-value than a later one.
Holm correction across metrics: Adjusted p-values are not made monotone · case 05
A metric with a smaller raw p-value reports a larger adjusted p-value than a later one.
Holm correction across metrics: Results are reported in sorted order · case 01
Rejection flags attach to the wrong metric names.
Holm correction across metrics: Results are reported in sorted order · case 02
Rejection flags attach to the wrong metric names.
Holm correction across metrics: Results are reported in sorted order · case 03
Rejection flags attach to the wrong metric names.
Holm correction across metrics: Results are reported in sorted order · case 04
Rejection flags attach to the wrong metric names.
Holm correction across metrics: Results are reported in sorted order · case 05
Rejection flags attach to the wrong metric names.
Per-arm sample size for conversion tests: Alpha is not split across tails · case 01
Planned sample sizes are too small for the two-sided test that is actually run.
Per-arm sample size for conversion tests: Alpha is not split across tails · case 02
Planned sample sizes are too small for the two-sided test that is actually run.
Per-arm sample size for conversion tests: Alpha is not split across tails · case 03
Planned sample sizes are too small for the two-sided test that is actually run.
Per-arm sample size for conversion tests: Alpha is not split across tails · case 04
Planned sample sizes are too small for the two-sided test that is actually run.
Per-arm sample size for conversion tests: Alpha is not split across tails · case 05
Planned sample sizes are too small for the two-sided test that is actually run.
INSPECTABLE BY DESIGN
Every result has a runnable source.
Runnable implementations with recorded outputs, source hashes, and explicit contracts. Related variants share a failure mechanism and belong together in evaluation splits.
Read the methodology ↗