Understand the failure.
Verify the repair.
Small, reproducible software failures. The broken implementation, the fix that didn’t work, and the one that passed—preserved together.
Explore the cases ↓How results are verified ↗WHAT THE ARCHIVE CONTAINS
100840 executable cases. 20168 are open.
Every case records the implementation that fails, the fix that did not work, and the repair that passed its checks—with recorded outputs and source hashes. This release adds 100840 cases across 20168 failure mechanisms and 254 domains.
The open tier gives you the failure and the unsuccessful fix for one case in every mechanism. The remaining 80672 cases, 5 variants per mechanism, are member-only: the verified repair, its recorded checks, and the full fixture suite are held in the member archive. Read the methodology ↗
A RECORD OF WHAT WENT WRONG
Browse the archive / 100840
Per-arm sample size for conversion tests: Relative effects are treated as absolute · case 01
A 10 percent relative lift on a 5 percent baseline is planned as a jump to 15 percent.
Per-arm sample size for conversion tests: Relative effects are treated as absolute · case 02
A 10 percent relative lift on a 5 percent baseline is planned as a jump to 15 percent.
Per-arm sample size for conversion tests: Relative effects are treated as absolute · case 03
A 10 percent relative lift on a 5 percent baseline is planned as a jump to 15 percent.
Per-arm sample size for conversion tests: Relative effects are treated as absolute · case 04
A 10 percent relative lift on a 5 percent baseline is planned as a jump to 15 percent.
Per-arm sample size for conversion tests: Relative effects are treated as absolute · case 05
A 10 percent relative lift on a 5 percent baseline is planned as a jump to 15 percent.
Per-arm sample size for conversion tests: Sample size is rounded to nearest · case 01
Planned samples fall just short of the requested power.
Per-arm sample size for conversion tests: Sample size is rounded to nearest · case 02
Planned samples fall just short of the requested power.
Per-arm sample size for conversion tests: Sample size is rounded to nearest · case 03
Planned samples fall just short of the requested power.
Per-arm sample size for conversion tests: Sample size is rounded to nearest · case 04
Planned samples fall just short of the requested power.
Per-arm sample size for conversion tests: Sample size is rounded to nearest · case 05
Planned samples fall just short of the requested power.
Per-arm sample size for conversion tests: Only the baseline variance is used · case 01
Plans ignore that the treatment rate has a different variance.
Per-arm sample size for conversion tests: Only the baseline variance is used · case 02
Plans ignore that the treatment rate has a different variance.
Per-arm sample size for conversion tests: Only the baseline variance is used · case 03
Plans ignore that the treatment rate has a different variance.
Per-arm sample size for conversion tests: Only the baseline variance is used · case 04
Plans ignore that the treatment rate has a different variance.
Per-arm sample size for conversion tests: Only the baseline variance is used · case 05
Plans ignore that the treatment rate has a different variance.
Per-arm sample size for conversion tests: The power quantile is taken from the wrong tail · case 01
Higher requested power produces smaller planned samples.
Per-arm sample size for conversion tests: The power quantile is taken from the wrong tail · case 02
Higher requested power produces smaller planned samples.
Per-arm sample size for conversion tests: The power quantile is taken from the wrong tail · case 03
Higher requested power produces smaller planned samples.
Per-arm sample size for conversion tests: The power quantile is taken from the wrong tail · case 04
Higher requested power produces smaller planned samples.
Per-arm sample size for conversion tests: The power quantile is taken from the wrong tail · case 05
Higher requested power produces smaller planned samples.
Per-user ratio metric variance: The metric averages per-user ratios · case 01
Light users with one session swing the metric as much as heavy users.
Per-user ratio metric variance: The metric averages per-user ratios · case 02
Light users with one session swing the metric as much as heavy users.
Per-user ratio metric variance: The metric averages per-user ratios · case 03
Light users with one session swing the metric as much as heavy users.
Per-user ratio metric variance: The metric averages per-user ratios · case 04
Light users with one session swing the metric as much as heavy users.
Per-user ratio metric variance: The metric averages per-user ratios · case 05
Light users with one session swing the metric as much as heavy users.
Per-user ratio metric variance: The covariance term is added · case 01
Standard errors are too large because positively correlated clicks and sessions no longer cancel.
Per-user ratio metric variance: The covariance term is added · case 02
Standard errors are too large because positively correlated clicks and sessions no longer cancel.
Per-user ratio metric variance: The covariance term is added · case 03
Standard errors are too large because positively correlated clicks and sessions no longer cancel.
Per-user ratio metric variance: The covariance term is added · case 04
Standard errors are too large because positively correlated clicks and sessions no longer cancel.
Per-user ratio metric variance: The covariance term is added · case 05
Standard errors are too large because positively correlated clicks and sessions no longer cancel.
Per-user ratio metric variance: Sessions are treated as independent trials · case 01
Intervals are far too narrow because repeated sessions of one user are assumed independent.
Per-user ratio metric variance: Sessions are treated as independent trials · case 02
Intervals are far too narrow because repeated sessions of one user are assumed independent.
Per-user ratio metric variance: Sessions are treated as independent trials · case 03
Intervals are far too narrow because repeated sessions of one user are assumed independent.
Per-user ratio metric variance: Sessions are treated as independent trials · case 04
Intervals are far too narrow because repeated sessions of one user are assumed independent.
Per-user ratio metric variance: Sessions are treated as independent trials · case 05
Intervals are far too narrow because repeated sessions of one user are assumed independent.
Per-user ratio metric variance: The delta-method variance is divided by mean sessions once · case 01
Standard errors scale with the session rate instead of its square.
Per-user ratio metric variance: The delta-method variance is divided by mean sessions once · case 02
Standard errors scale with the session rate instead of its square.
Per-user ratio metric variance: The delta-method variance is divided by mean sessions once · case 03
Standard errors scale with the session rate instead of its square.
Per-user ratio metric variance: The delta-method variance is divided by mean sessions once · case 04
Standard errors scale with the session rate instead of its square.
Per-user ratio metric variance: The delta-method variance is divided by mean sessions once · case 05
Standard errors scale with the session rate instead of its square.
Post-exposure metric window: Activity before exposure is counted · case 01
Treatment effects are diluted by behaviour that happened before users saw the change.
Post-exposure metric window: Activity before exposure is counted · case 02
Treatment effects are diluted by behaviour that happened before users saw the change.
Post-exposure metric window: Activity before exposure is counted · case 03
Treatment effects are diluted by behaviour that happened before users saw the change.
Post-exposure metric window: Activity before exposure is counted · case 04
Treatment effects are diluted by behaviour that happened before users saw the change.
Post-exposure metric window: Activity before exposure is counted · case 05
Treatment effects are diluted by behaviour that happened before users saw the change.
Post-exposure metric window: The analysis window includes one extra day · case 01
A 7-day metric actually sums 8 days of activity.
Post-exposure metric window: The analysis window includes one extra day · case 02
A 7-day metric actually sums 8 days of activity.
Post-exposure metric window: The analysis window includes one extra day · case 03
A 7-day metric actually sums 8 days of activity.
Post-exposure metric window: The analysis window includes one extra day · case 04
A 7-day metric actually sums 8 days of activity.
Post-exposure metric window: The analysis window includes one extra day · case 05
A 7-day metric actually sums 8 days of activity.
Post-exposure metric window: Exposed users without events disappear · case 01
Metric means are inflated because inactive exposed users are excluded.
Post-exposure metric window: Exposed users without events disappear · case 02
Metric means are inflated because inactive exposed users are excluded.
Post-exposure metric window: Exposed users without events disappear · case 03
Metric means are inflated because inactive exposed users are excluded.
Post-exposure metric window: Exposed users without events disappear · case 04
Metric means are inflated because inactive exposed users are excluded.
Post-exposure metric window: Exposed users without events disappear · case 05
Metric means are inflated because inactive exposed users are excluded.
Post-exposure metric window: All users share the earliest exposure as window start · case 01
Late-exposed users contribute almost nothing because their window closed before they arrived.
Post-exposure metric window: All users share the earliest exposure as window start · case 02
Late-exposed users contribute almost nothing because their window closed before they arrived.
Post-exposure metric window: All users share the earliest exposure as window start · case 03
Late-exposed users contribute almost nothing because their window closed before they arrived.
Post-exposure metric window: All users share the earliest exposure as window start · case 04
Late-exposed users contribute almost nothing because their window closed before they arrived.
Post-exposure metric window: All users share the earliest exposure as window start · case 05
Late-exposed users contribute almost nothing because their window closed before they arrived.
Revenue outlier capping: The cap is computed from the control arm alone · case 01
A treatment that creates more big spenders gets its gains clipped by a control-based cap.
Revenue outlier capping: The cap is computed from the control arm alone · case 02
A treatment that creates more big spenders gets its gains clipped by a control-based cap.
Revenue outlier capping: The cap is computed from the control arm alone · case 03
A treatment that creates more big spenders gets its gains clipped by a control-based cap.
Revenue outlier capping: The cap is computed from the control arm alone · case 04
A treatment that creates more big spenders gets its gains clipped by a control-based cap.
Revenue outlier capping: The cap is computed from the control arm alone · case 05
A treatment that creates more big spenders gets its gains clipped by a control-based cap.
Revenue outlier capping: The cap reads one rank too high · case 01
The cap sits one order statistic above the requested percentile.
Revenue outlier capping: The cap reads one rank too high · case 02
The cap sits one order statistic above the requested percentile.
Revenue outlier capping: The cap reads one rank too high · case 03
The cap sits one order statistic above the requested percentile.
Revenue outlier capping: The cap reads one rank too high · case 04
The cap sits one order statistic above the requested percentile.
Revenue outlier capping: The cap reads one rank too high · case 05
The cap sits one order statistic above the requested percentile.
Revenue outlier capping: Outliers are dropped instead of capped · case 01
Revenue means understate treatment impact because whales contribute nothing.
Revenue outlier capping: Outliers are dropped instead of capped · case 02
Revenue means understate treatment impact because whales contribute nothing.
Revenue outlier capping: Outliers are dropped instead of capped · case 03
Revenue means understate treatment impact because whales contribute nothing.
Revenue outlier capping: Outliers are dropped instead of capped · case 04
Revenue means understate treatment impact because whales contribute nothing.
Revenue outlier capping: Outliers are dropped instead of capped · case 05
Revenue means understate treatment impact because whales contribute nothing.
Revenue outlier capping: Percentile rank is rounded to nearest · case 01
Small experiments cap at a lower order statistic than the nearest-rank rule allows.
Revenue outlier capping: Percentile rank is rounded to nearest · case 02
Small experiments cap at a lower order statistic than the nearest-rank rule allows.
Revenue outlier capping: Percentile rank is rounded to nearest · case 03
Small experiments cap at a lower order statistic than the nearest-rank rule allows.
Revenue outlier capping: Percentile rank is rounded to nearest · case 04
Small experiments cap at a lower order statistic than the nearest-rank rule allows.
Revenue outlier capping: Percentile rank is rounded to nearest · case 05
Small experiments cap at a lower order statistic than the nearest-rank rule allows.
Bayesian conversion comparison: Posterior beta ignores observed successes · case 01
Posterior means are too low because every trial is counted as a failure.
Bayesian conversion comparison: Posterior beta ignores observed successes · case 02
Posterior means are too low because every trial is counted as a failure.
Bayesian conversion comparison: Posterior beta ignores observed successes · case 03
Posterior means are too low because every trial is counted as a failure.
Bayesian conversion comparison: Posterior beta ignores observed successes · case 04
Posterior means are too low because every trial is counted as a failure.
Bayesian conversion comparison: Posterior beta ignores observed successes · case 05
Posterior means are too low because every trial is counted as a failure.
Bayesian conversion comparison: Posterior means report raw conversion rates · case 01
Displayed means ignore the prior, contradicting the reported probability.
Bayesian conversion comparison: Posterior means report raw conversion rates · case 02
Displayed means ignore the prior, contradicting the reported probability.
Bayesian conversion comparison: Posterior means report raw conversion rates · case 03
Displayed means ignore the prior, contradicting the reported probability.
Bayesian conversion comparison: Posterior means report raw conversion rates · case 04
Displayed means ignore the prior, contradicting the reported probability.
Bayesian conversion comparison: Posterior means report raw conversion rates · case 05
Displayed means ignore the prior, contradicting the reported probability.
Bayesian conversion comparison: The probability sum stops one term early · case 01
Chance-to-beat-control is systematically understated.
Bayesian conversion comparison: The probability sum stops one term early · case 02
Chance-to-beat-control is systematically understated.
Bayesian conversion comparison: The probability sum stops one term early · case 03
Chance-to-beat-control is systematically understated.
Bayesian conversion comparison: The probability sum stops one term early · case 04
Chance-to-beat-control is systematically understated.
Bayesian conversion comparison: The probability sum stops one term early · case 05
Chance-to-beat-control is systematically understated.
Bayesian conversion comparison: The sum is normalised by the treatment posterior · case 01
Probabilities can exceed one or fall far below the truth.
Bayesian conversion comparison: The sum is normalised by the treatment posterior · case 02
Probabilities can exceed one or fall far below the truth.
Bayesian conversion comparison: The sum is normalised by the treatment posterior · case 03
Probabilities can exceed one or fall far below the truth.
Bayesian conversion comparison: The sum is normalised by the treatment posterior · case 04
Probabilities can exceed one or fall far below the truth.
Bayesian conversion comparison: The sum is normalised by the treatment posterior · case 05
Probabilities can exceed one or fall far below the truth.
INSPECTABLE BY DESIGN
Every result has a runnable source.
Runnable implementations with recorded outputs, source hashes, and explicit contracts. Related variants share a failure mechanism and belong together in evaluation splits.
Read the methodology ↗