Understand the failure.
Verify the repair.
Small, reproducible software failures. The broken implementation, the fix that didn’t work, and the one that passed—preserved together.
Explore the cases ↓How results are verified ↗WHAT THE ARCHIVE CONTAINS
100840 executable cases. 20168 are open.
Every case records the implementation that fails, the fix that did not work, and the repair that passed its checks—with recorded outputs and source hashes. This release adds 100840 cases across 20168 failure mechanisms and 254 domains.
The open tier gives you the failure and the unsuccessful fix for one case in every mechanism. The remaining 80672 cases, 5 variants per mechanism, are member-only: the verified repair, its recorded checks, and the full fixture suite are held in the member archive. Read the methodology ↗
A RECORD OF WHAT WENT WRONG
Browse the archive / 100840
Bayesian conversion comparison: The per-term denominator drops the index · case 01
Each term is scaled by 1/beta_B instead of 1/(beta_B + i).
Bayesian conversion comparison: The per-term denominator drops the index · case 02
Each term is scaled by 1/beta_B instead of 1/(beta_B + i).
Bayesian conversion comparison: The per-term denominator drops the index · case 03
Each term is scaled by 1/beta_B instead of 1/(beta_B + i).
Bayesian conversion comparison: The per-term denominator drops the index · case 04
Each term is scaled by 1/beta_B instead of 1/(beta_B + i).
Bayesian conversion comparison: The per-term denominator drops the index · case 05
Each term is scaled by 1/beta_B instead of 1/(beta_B + i).
Triggered effect dilution: The trigger rate comes from the treatment arm · case 01
A treatment that changes who triggers distorts the diluted estimate.
Triggered effect dilution: The trigger rate comes from the treatment arm · case 02
A treatment that changes who triggers distorts the diluted estimate.
Triggered effect dilution: The trigger rate comes from the treatment arm · case 03
A treatment that changes who triggers distorts the diluted estimate.
Triggered effect dilution: The trigger rate comes from the treatment arm · case 04
A treatment that changes who triggers distorts the diluted estimate.
Triggered effect dilution: The trigger rate comes from the treatment arm · case 05
A treatment that changes who triggers distorts the diluted estimate.
Triggered effect dilution: Relative impact uses the undiluted effect · case 01
Site-wide relative lift is overstated by the inverse trigger rate.
Triggered effect dilution: Relative impact uses the undiluted effect · case 02
Site-wide relative lift is overstated by the inverse trigger rate.
Triggered effect dilution: Relative impact uses the undiluted effect · case 03
Site-wide relative lift is overstated by the inverse trigger rate.
Triggered effect dilution: Relative impact uses the undiluted effect · case 04
Site-wide relative lift is overstated by the inverse trigger rate.
Triggered effect dilution: Relative impact uses the undiluted effect · case 05
Site-wide relative lift is overstated by the inverse trigger rate.
Triggered effect dilution: The triggered effect is scaled up instead of down · case 01
Reported site-wide gains exceed the triggered gains.
Triggered effect dilution: The triggered effect is scaled up instead of down · case 02
Reported site-wide gains exceed the triggered gains.
Triggered effect dilution: The triggered effect is scaled up instead of down · case 03
Reported site-wide gains exceed the triggered gains.
Triggered effect dilution: The triggered effect is scaled up instead of down · case 04
Reported site-wide gains exceed the triggered gains.
Triggered effect dilution: The triggered effect is scaled up instead of down · case 05
Reported site-wide gains exceed the triggered gains.
Guardrailed ship decision: Lower-is-better guardrails check the lower bound · case 01
A latency regression whose interval reaches far above the margin still passes.
Guardrailed ship decision: Lower-is-better guardrails check the lower bound · case 02
A latency regression whose interval reaches far above the margin still passes.
Guardrailed ship decision: Lower-is-better guardrails check the lower bound · case 03
A latency regression whose interval reaches far above the margin still passes.
Guardrailed ship decision: Lower-is-better guardrails check the lower bound · case 04
A latency regression whose interval reaches far above the margin still passes.
Guardrailed ship decision: Lower-is-better guardrails check the lower bound · case 05
A latency regression whose interval reaches far above the margin still passes.
Guardrailed ship decision: Non-inferiority margin has the wrong sign · case 01
Higher-is-better guardrails demand a gain larger than the margin.
Guardrailed ship decision: Non-inferiority margin has the wrong sign · case 02
Higher-is-better guardrails demand a gain larger than the margin.
Guardrailed ship decision: Non-inferiority margin has the wrong sign · case 03
Higher-is-better guardrails demand a gain larger than the margin.
Guardrailed ship decision: Non-inferiority margin has the wrong sign · case 04
Higher-is-better guardrails demand a gain larger than the margin.
Guardrailed ship decision: Non-inferiority margin has the wrong sign · case 05
Higher-is-better guardrails demand a gain larger than the margin.
Guardrailed ship decision: A bound exactly at the margin passes · case 01
A guardrail whose lower bound equals the tolerated loss is accepted.
Guardrailed ship decision: A bound exactly at the margin passes · case 02
A guardrail whose lower bound equals the tolerated loss is accepted.
Guardrailed ship decision: A bound exactly at the margin passes · case 03
A guardrail whose lower bound equals the tolerated loss is accepted.
Guardrailed ship decision: A bound exactly at the margin passes · case 04
A guardrail whose lower bound equals the tolerated loss is accepted.
Guardrailed ship decision: A bound exactly at the margin passes · case 05
A guardrail whose lower bound equals the tolerated loss is accepted.
Guardrailed ship decision: The last failing guardrail is reported · case 01
Reviews chase a minor guardrail while the first listed blocker is hidden.
Guardrailed ship decision: The last failing guardrail is reported · case 02
Reviews chase a minor guardrail while the first listed blocker is hidden.
Guardrailed ship decision: The last failing guardrail is reported · case 03
Reviews chase a minor guardrail while the first listed blocker is hidden.
Guardrailed ship decision: The last failing guardrail is reported · case 04
Reviews chase a minor guardrail while the first listed blocker is hidden.
Guardrailed ship decision: The last failing guardrail is reported · case 05
Reviews chase a minor guardrail while the first listed blocker is hidden.
Guardrailed ship decision: A positive point estimate counts as a win · case 01
Launches ship on noise whenever the lift is positive.
Guardrailed ship decision: A positive point estimate counts as a win · case 02
Launches ship on noise whenever the lift is positive.
Guardrailed ship decision: A positive point estimate counts as a win · case 03
Launches ship on noise whenever the lift is positive.
Guardrailed ship decision: A positive point estimate counts as a win · case 04
Launches ship on noise whenever the lift is positive.
Guardrailed ship decision: A positive point estimate counts as a win · case 05
Launches ship on noise whenever the lift is positive.
Bootstrap difference interval: Resampling draws without replacement · case 01
Every replicate reproduces the observed difference, so the interval has zero width.
Bootstrap difference interval: Resampling draws without replacement · case 02
Every replicate reproduces the observed difference, so the interval has zero width.
Bootstrap difference interval: Resampling draws without replacement · case 03
Every replicate reproduces the observed difference, so the interval has zero width.
Bootstrap difference interval: Resampling draws without replacement · case 04
Every replicate reproduces the observed difference, so the interval has zero width.
Bootstrap difference interval: Resampling draws without replacement · case 05
Every replicate reproduces the observed difference, so the interval has zero width.
Bootstrap difference interval: The upper percentile reads one position too far · case 01
Upper bounds are systematically too high.
Bootstrap difference interval: The upper percentile reads one position too far · case 02
Upper bounds are systematically too high.
Bootstrap difference interval: The upper percentile reads one position too far · case 03
Upper bounds are systematically too high.
Bootstrap difference interval: The upper percentile reads one position too far · case 04
Upper bounds are systematically too high.
Bootstrap difference interval: The upper percentile reads one position too far · case 05
Upper bounds are systematically too high.
Bootstrap difference interval: The lower tail is not halved · case 01
Lower bounds sit at twice the intended tail probability.
Bootstrap difference interval: The lower tail is not halved · case 02
Lower bounds sit at twice the intended tail probability.
Bootstrap difference interval: The lower tail is not halved · case 03
Lower bounds sit at twice the intended tail probability.
Bootstrap difference interval: The lower tail is not halved · case 04
Lower bounds sit at twice the intended tail probability.
Bootstrap difference interval: The lower tail is not halved · case 05
Lower bounds sit at twice the intended tail probability.
Bootstrap difference interval: Replicates record control minus treatment · case 01
The interval is mirrored, so improvements appear as regressions.
Bootstrap difference interval: Replicates record control minus treatment · case 02
The interval is mirrored, so improvements appear as regressions.
Bootstrap difference interval: Replicates record control minus treatment · case 03
The interval is mirrored, so improvements appear as regressions.
Bootstrap difference interval: Replicates record control minus treatment · case 04
The interval is mirrored, so improvements appear as regressions.
Bootstrap difference interval: Replicates record control minus treatment · case 05
The interval is mirrored, so improvements appear as regressions.
Bootstrap difference interval: Treatment is resampled before control · case 01
Intervals are not reproducible against the reference implementation for the same seed.
Bootstrap difference interval: Treatment is resampled before control · case 02
Intervals are not reproducible against the reference implementation for the same seed.
Bootstrap difference interval: Treatment is resampled before control · case 03
Intervals are not reproducible against the reference implementation for the same seed.
Bootstrap difference interval: Treatment is resampled before control · case 04
Intervals are not reproducible against the reference implementation for the same seed.
Bootstrap difference interval: Treatment is resampled before control · case 05
Intervals are not reproducible against the reference implementation for the same seed.
Rank-sum test with ties: Tied values receive ordinal ranks · case 01
The statistic depends on arbitrary ordering of tied values and favours whichever arm sorts first.
Rank-sum test with ties: Tied values receive ordinal ranks · case 02
The statistic depends on arbitrary ordering of tied values and favours whichever arm sorts first.
Rank-sum test with ties: Tied values receive ordinal ranks · case 03
The statistic depends on arbitrary ordering of tied values and favours whichever arm sorts first.
Rank-sum test with ties: Tied values receive ordinal ranks · case 04
The statistic depends on arbitrary ordering of tied values and favours whichever arm sorts first.
Rank-sum test with ties: Tied values receive ordinal ranks · case 05
The statistic depends on arbitrary ordering of tied values and favours whichever arm sorts first.
Rank-sum test with ties: Variance ignores ties · case 01
z-scores are too small on bucketed metrics with many ties.
Rank-sum test with ties: Variance ignores ties · case 02
z-scores are too small on bucketed metrics with many ties.
Rank-sum test with ties: Variance ignores ties · case 03
z-scores are too small on bucketed metrics with many ties.
Rank-sum test with ties: Variance ignores ties · case 04
z-scores are too small on bucketed metrics with many ties.
Rank-sum test with ties: Variance ignores ties · case 05
z-scores are too small on bucketed metrics with many ties.
Rank-sum test with ties: U subtracts the control arm offset · case 01
U is shifted whenever the arms have different sizes.
Rank-sum test with ties: U subtracts the control arm offset · case 02
U is shifted whenever the arms have different sizes.
Rank-sum test with ties: U subtracts the control arm offset · case 03
U is shifted whenever the arms have different sizes.
Rank-sum test with ties: U subtracts the control arm offset · case 04
U is shifted whenever the arms have different sizes.
Rank-sum test with ties: U subtracts the control arm offset · case 05
U is shifted whenever the arms have different sizes.
Rank-sum test with ties: z is centred on the wrong null mean · case 01
Balanced data produces strongly negative z-scores.
Rank-sum test with ties: z is centred on the wrong null mean · case 02
Balanced data produces strongly negative z-scores.
Rank-sum test with ties: z is centred on the wrong null mean · case 03
Balanced data produces strongly negative z-scores.
Rank-sum test with ties: z is centred on the wrong null mean · case 04
Balanced data produces strongly negative z-scores.
Rank-sum test with ties: z is centred on the wrong null mean · case 05
Balanced data produces strongly negative z-scores.
Complier average causal effect: Treatment outcomes are averaged over adopters · case 01
The ITT estimate mixes a per-protocol treatment mean with an intent-to-treat control mean.
Complier average causal effect: Treatment outcomes are averaged over adopters · case 02
The ITT estimate mixes a per-protocol treatment mean with an intent-to-treat control mean.
Complier average causal effect: Treatment outcomes are averaged over adopters · case 03
The ITT estimate mixes a per-protocol treatment mean with an intent-to-treat control mean.
Complier average causal effect: Treatment outcomes are averaged over adopters · case 04
The ITT estimate mixes a per-protocol treatment mean with an intent-to-treat control mean.
Complier average causal effect: Treatment outcomes are averaged over adopters · case 05
The ITT estimate mixes a per-protocol treatment mean with an intent-to-treat control mean.
Complier average causal effect: Control crossover is ignored in compliance · case 01
CACE is understated when control users can reach the feature through another path.
Complier average causal effect: Control crossover is ignored in compliance · case 02
CACE is understated when control users can reach the feature through another path.
Complier average causal effect: Control crossover is ignored in compliance · case 03
CACE is understated when control users can reach the feature through another path.
Complier average causal effect: Control crossover is ignored in compliance · case 04
CACE is understated when control users can reach the feature through another path.
Complier average causal effect: Control crossover is ignored in compliance · case 05
CACE is understated when control users can reach the feature through another path.
INSPECTABLE BY DESIGN
Every result has a runnable source.
Runnable implementations with recorded outputs, source hashes, and explicit contracts. Related variants share a failure mechanism and belong together in evaluation splits.
Read the methodology ↗