Understand the failure.
Verify the repair.
Small, reproducible software failures. The broken implementation, the fix that didn’t work, and the one that passed—preserved together.
Explore the cases ↓How results are verified ↗WHAT THE ARCHIVE CONTAINS
100840 executable cases. 20168 are open.
Every case records the implementation that fails, the fix that did not work, and the repair that passed its checks—with recorded outputs and source hashes. This release adds 100840 cases across 20168 failure mechanisms and 254 domains.
The open tier gives you the failure and the unsuccessful fix for one case in every mechanism. The remaining 80672 cases, 5 variants per mechanism, are member-only: the verified repair, its recorded checks, and the full fixture suite are held in the member archive. Read the methodology ↗
A RECORD OF WHAT WENT WRONG
Browse the archive / 100840
Complier average causal effect: Negative compliance differences produce a CACE · case 01
A treatment that lowered uptake reports an inverted complier effect.
Complier average causal effect: Negative compliance differences produce a CACE · case 02
A treatment that lowered uptake reports an inverted complier effect.
Complier average causal effect: Negative compliance differences produce a CACE · case 03
A treatment that lowered uptake reports an inverted complier effect.
Complier average causal effect: Negative compliance differences produce a CACE · case 04
A treatment that lowered uptake reports an inverted complier effect.
Complier average causal effect: Negative compliance differences produce a CACE · case 05
A treatment that lowered uptake reports an inverted complier effect.
Cluster randomisation design effect: Unequal cluster sizes are ignored · case 01
Design effects are understated when a few accounts are much larger than the rest.
Cluster randomisation design effect: Unequal cluster sizes are ignored · case 02
Design effects are understated when a few accounts are much larger than the rest.
Cluster randomisation design effect: Unequal cluster sizes are ignored · case 03
Design effects are understated when a few accounts are much larger than the rest.
Cluster randomisation design effect: Unequal cluster sizes are ignored · case 04
Design effects are understated when a few accounts are much larger than the rest.
Cluster randomisation design effect: Unequal cluster sizes are ignored · case 05
Design effects are understated when a few accounts are much larger than the rest.
Cluster randomisation design effect: Effective sample is multiplied by the design effect · case 01
Clustering appears to increase information instead of reducing it.
Cluster randomisation design effect: Effective sample is multiplied by the design effect · case 02
Clustering appears to increase information instead of reducing it.
Cluster randomisation design effect: Effective sample is multiplied by the design effect · case 03
Clustering appears to increase information instead of reducing it.
Cluster randomisation design effect: Effective sample is multiplied by the design effect · case 04
Clustering appears to increase information instead of reducing it.
Cluster randomisation design effect: Effective sample is multiplied by the design effect · case 05
Clustering appears to increase information instead of reducing it.
Cluster randomisation design effect: Required clusters omit the design effect · case 01
Recruitment targets are too small to reach the planned power.
Cluster randomisation design effect: Required clusters omit the design effect · case 02
Recruitment targets are too small to reach the planned power.
Cluster randomisation design effect: Required clusters omit the design effect · case 03
Recruitment targets are too small to reach the planned power.
Cluster randomisation design effect: Required clusters omit the design effect · case 04
Recruitment targets are too small to reach the planned power.
Cluster randomisation design effect: Required clusters omit the design effect · case 05
Recruitment targets are too small to reach the planned power.
Cluster randomisation design effect: Cluster size variance uses a sample divisor · case 01
Design effects for a fixed set of clusters are inflated.
Cluster randomisation design effect: Cluster size variance uses a sample divisor · case 02
Design effects for a fixed set of clusters are inflated.
Cluster randomisation design effect: Cluster size variance uses a sample divisor · case 03
Design effects for a fixed set of clusters are inflated.
Cluster randomisation design effect: Cluster size variance uses a sample divisor · case 04
Design effects for a fixed set of clusters are inflated.
Cluster randomisation design effect: Cluster size variance uses a sample divisor · case 05
Design effects for a fixed set of clusters are inflated.
O'Brien-Fleming type alpha spending: Cumulative spend is reported as incremental · case 01
Later looks appear to have almost the entire alpha available again.
O'Brien-Fleming type alpha spending: Cumulative spend is reported as incremental · case 02
Later looks appear to have almost the entire alpha available again.
O'Brien-Fleming type alpha spending: Cumulative spend is reported as incremental · case 03
Later looks appear to have almost the entire alpha available again.
O'Brien-Fleming type alpha spending: Cumulative spend is reported as incremental · case 04
Later looks appear to have almost the entire alpha available again.
O'Brien-Fleming type alpha spending: Cumulative spend is reported as incremental · case 05
Later looks appear to have almost the entire alpha available again.
O'Brien-Fleming type alpha spending: The spending boundary uses a one-sided quantile · case 01
The total alpha spent by the final look is double the design alpha.
O'Brien-Fleming type alpha spending: The spending boundary uses a one-sided quantile · case 02
The total alpha spent by the final look is double the design alpha.
O'Brien-Fleming type alpha spending: The spending boundary uses a one-sided quantile · case 03
The total alpha spent by the final look is double the design alpha.
O'Brien-Fleming type alpha spending: The spending boundary uses a one-sided quantile · case 04
The total alpha spent by the final look is double the design alpha.
O'Brien-Fleming type alpha spending: The spending boundary uses a one-sided quantile · case 05
The total alpha spent by the final look is double the design alpha.
O'Brien-Fleming type alpha spending: Boundary scales with t instead of sqrt(t) · case 01
Early looks spend far too little alpha and late looks too much.
O'Brien-Fleming type alpha spending: Boundary scales with t instead of sqrt(t) · case 02
Early looks spend far too little alpha and late looks too much.
O'Brien-Fleming type alpha spending: Boundary scales with t instead of sqrt(t) · case 03
Early looks spend far too little alpha and late looks too much.
O'Brien-Fleming type alpha spending: Boundary scales with t instead of sqrt(t) · case 04
Early looks spend far too little alpha and late looks too much.
O'Brien-Fleming type alpha spending: Boundary scales with t instead of sqrt(t) · case 05
Early looks spend far too little alpha and late looks too much.
O'Brien-Fleming type alpha spending: Repeated information fractions are accepted · case 01
A duplicated look slips through validation and reports zero alpha spent.
O'Brien-Fleming type alpha spending: Repeated information fractions are accepted · case 02
A duplicated look slips through validation and reports zero alpha spent.
O'Brien-Fleming type alpha spending: Repeated information fractions are accepted · case 03
A duplicated look slips through validation and reports zero alpha spent.
O'Brien-Fleming type alpha spending: Repeated information fractions are accepted · case 04
A duplicated look slips through validation and reports zero alpha spent.
O'Brien-Fleming type alpha spending: Repeated information fractions are accepted · case 05
A duplicated look slips through validation and reports zero alpha spent.
Pre-period balance check: The difference is scaled by the control spread only · case 01
A treatment arm with extra variance changes the balance score asymmetrically.
Pre-period balance check: The difference is scaled by the control spread only · case 02
A treatment arm with extra variance changes the balance score asymmetrically.
Pre-period balance check: The difference is scaled by the control spread only · case 03
A treatment arm with extra variance changes the balance score asymmetrically.
Pre-period balance check: The difference is scaled by the control spread only · case 04
A treatment arm with extra variance changes the balance score asymmetrically.
Pre-period balance check: The difference is scaled by the control spread only · case 05
A treatment arm with extra variance changes the balance score asymmetrically.
Pre-period balance check: Balance uses population variances · case 01
Tiny pre-period samples exaggerate imbalance.
Pre-period balance check: Balance uses population variances · case 02
Tiny pre-period samples exaggerate imbalance.
Pre-period balance check: Balance uses population variances · case 03
Tiny pre-period samples exaggerate imbalance.
Pre-period balance check: Balance uses population variances · case 04
Tiny pre-period samples exaggerate imbalance.
Pre-period balance check: Balance uses population variances · case 05
Tiny pre-period samples exaggerate imbalance.
Pre-period balance check: Only positive imbalance is flagged · case 01
Treatment arms that start lower than control pass the balance check.
Pre-period balance check: Only positive imbalance is flagged · case 02
Treatment arms that start lower than control pass the balance check.
Pre-period balance check: Only positive imbalance is flagged · case 03
Treatment arms that start lower than control pass the balance check.
Pre-period balance check: Only positive imbalance is flagged · case 04
Treatment arms that start lower than control pass the balance check.
Pre-period balance check: Only positive imbalance is flagged · case 05
Treatment arms that start lower than control pass the balance check.
Pre-period balance check: Constant equal arms are reported as imbalanced · case 01
A binary pre-period flag that is identical in both arms blocks the readout.
Pre-period balance check: Constant equal arms are reported as imbalanced · case 02
A binary pre-period flag that is identical in both arms blocks the readout.
Pre-period balance check: Constant equal arms are reported as imbalanced · case 03
A binary pre-period flag that is identical in both arms blocks the readout.
Pre-period balance check: Constant equal arms are reported as imbalanced · case 04
A binary pre-period flag that is identical in both arms blocks the readout.
Pre-period balance check: Constant equal arms are reported as imbalanced · case 05
A binary pre-period flag that is identical in both arms blocks the readout.
Team-draft interleaving credit: Team B can place a document already shown · case 01
The interleaved page shows the same result twice and B gets extra credit.
Team-draft interleaving credit: Team B can place a document already shown · case 02
The interleaved page shows the same result twice and B gets extra credit.
Team-draft interleaving credit: Team B can place a document already shown · case 03
The interleaved page shows the same result twice and B gets extra credit.
Team-draft interleaving credit: Team B can place a document already shown · case 04
The interleaved page shows the same result twice and B gets extra credit.
Team-draft interleaving credit: Team B can place a document already shown · case 05
The interleaved page shows the same result twice and B gets extra credit.
Team-draft interleaving credit: A coin is consumed every round · case 01
Tie-break draws no longer follow the logged coin sequence, so replays disagree.
Team-draft interleaving credit: A coin is consumed every round · case 02
Tie-break draws no longer follow the logged coin sequence, so replays disagree.
Team-draft interleaving credit: A coin is consumed every round · case 03
Tie-break draws no longer follow the logged coin sequence, so replays disagree.
Team-draft interleaving credit: A coin is consumed every round · case 04
Tie-break draws no longer follow the logged coin sequence, so replays disagree.
Team-draft interleaving credit: A coin is consumed every round · case 05
Tie-break draws no longer follow the logged coin sequence, so replays disagree.
Team-draft interleaving credit: The team ahead picks again · case 01
One ranker dominates the interleaved list whenever it wins the first coin.
Team-draft interleaving credit: The team ahead picks again · case 02
One ranker dominates the interleaved list whenever it wins the first coin.
Team-draft interleaving credit: The team ahead picks again · case 03
One ranker dominates the interleaved list whenever it wins the first coin.
Team-draft interleaving credit: The team ahead picks again · case 04
One ranker dominates the interleaved list whenever it wins the first coin.
Team-draft interleaving credit: The team ahead picks again · case 05
One ranker dominates the interleaved list whenever it wins the first coin.
Team-draft interleaving credit: Clicks credit every ranker containing the document · case 01
Clicks on documents both rankers returned count for both, masking real preferences.
Team-draft interleaving credit: Clicks credit every ranker containing the document · case 02
Clicks on documents both rankers returned count for both, masking real preferences.
Team-draft interleaving credit: Clicks credit every ranker containing the document · case 03
Clicks on documents both rankers returned count for both, masking real preferences.
Team-draft interleaving credit: Clicks credit every ranker containing the document · case 04
Clicks on documents both rankers returned count for both, masking real preferences.
Team-draft interleaving credit: Clicks credit every ranker containing the document · case 05
Clicks on documents both rankers returned count for both, masking real preferences.
Always-valid sequential p-value: The p-value can rise again · case 01
A result that was significant yesterday becomes insignificant today, breaking always-valid guarantees.
Always-valid sequential p-value: The p-value can rise again · case 02
A result that was significant yesterday becomes insignificant today, breaking always-valid guarantees.
Always-valid sequential p-value: The p-value can rise again · case 03
A result that was significant yesterday becomes insignificant today, breaking always-valid guarantees.
Always-valid sequential p-value: The p-value can rise again · case 04
A result that was significant yesterday becomes insignificant today, breaking always-valid guarantees.
Always-valid sequential p-value: The p-value can rise again · case 05
A result that was significant yesterday becomes insignificant today, breaking always-valid guarantees.
Always-valid sequential p-value: The exponent grows linearly in n · case 01
Evidence accumulates too slowly, delaying every decision.
Always-valid sequential p-value: The exponent grows linearly in n · case 02
Evidence accumulates too slowly, delaying every decision.
Always-valid sequential p-value: The exponent grows linearly in n · case 03
Evidence accumulates too slowly, delaying every decision.
Always-valid sequential p-value: The exponent grows linearly in n · case 04
Evidence accumulates too slowly, delaying every decision.
Always-valid sequential p-value: The exponent grows linearly in n · case 05
Evidence accumulates too slowly, delaying every decision.
Always-valid sequential p-value: The prefactor ignores accumulated sample size · case 01
Likelihood ratios are too large for long streams, overstating significance.
Always-valid sequential p-value: The prefactor ignores accumulated sample size · case 02
Likelihood ratios are too large for long streams, overstating significance.
Always-valid sequential p-value: The prefactor ignores accumulated sample size · case 03
Likelihood ratios are too large for long streams, overstating significance.
Always-valid sequential p-value: The prefactor ignores accumulated sample size · case 04
Likelihood ratios are too large for long streams, overstating significance.
Always-valid sequential p-value: The prefactor ignores accumulated sample size · case 05
Likelihood ratios are too large for long streams, overstating significance.
INSPECTABLE BY DESIGN
Every result has a runnable source.
Runnable implementations with recorded outputs, source hashes, and explicit contracts. Related variants share a failure mechanism and belong together in evaluation splits.
Read the methodology ↗