Understand the failure.
Verify the repair.
Small, reproducible software failures. The broken implementation, the fix that didn’t work, and the one that passed—preserved together.
Explore the cases ↓How results are verified ↗WHAT THE ARCHIVE CONTAINS
100840 executable cases. 20168 are open.
Every case records the implementation that fails, the fix that did not work, and the repair that passed its checks—with recorded outputs and source hashes. This release adds 100840 cases across 20168 failure mechanisms and 254 domains.
The open tier gives you the failure and the unsuccessful fix for one case in every mechanism. The remaining 80672 cases, 5 variants per mechanism, are member-only: the verified repair, its recorded checks, and the full fixture suite are held in the member archive. Read the methodology ↗
A RECORD OF WHAT WENT WRONG
Browse the archive / 100840
Timestamp targeting clause: Offset timestamps are read as UTC · case 01
A clause user value of 23:13:20+01:00 is treated as 23:13:20 UTC, an hour late.
Timestamp targeting clause: Offset timestamps are read as UTC · case 02
A clause user value of 23:13:20+01:00 is treated as 23:13:20 UTC, an hour late.
Timestamp targeting clause: Offset timestamps are read as UTC · case 03
A clause user value of 23:13:20+01:00 is treated as 23:13:20 UTC, an hour late.
Timestamp targeting clause: Offset timestamps are read as UTC · case 04
A clause user value of 23:13:20+01:00 is treated as 23:13:20 UTC, an hour late.
Timestamp targeting clause: Offset timestamps are read as UTC · case 05
A clause user value of 23:13:20+01:00 is treated as 23:13:20 UTC, an hour late.
Seeded weighted rollout: Seed zero is ignored · case 01
Flags configured with seed 0 bucket by flag key and salt, breaking the shared cohort.
Seeded weighted rollout: Seed zero is ignored · case 02
Flags configured with seed 0 bucket by flag key and salt, breaking the shared cohort.
Seeded weighted rollout: Seed zero is ignored · case 03
Flags configured with seed 0 bucket by flag key and salt, breaking the shared cohort.
Seeded weighted rollout: Seed zero is ignored · case 04
Flags configured with seed 0 bucket by flag key and salt, breaking the shared cohort.
Seeded weighted rollout: Seed zero is ignored · case 05
Flags configured with seed 0 bucket by flag key and salt, breaking the shared cohort.
Seeded weighted rollout: Empty secondary keys are dropped · case 01
Users with an empty secondary attribute bucket as if they had none.
Seeded weighted rollout: Empty secondary keys are dropped · case 02
Users with an empty secondary attribute bucket as if they had none.
Seeded weighted rollout: Empty secondary keys are dropped · case 03
Users with an empty secondary attribute bucket as if they had none.
Seeded weighted rollout: Empty secondary keys are dropped · case 04
Users with an empty secondary attribute bucket as if they had none.
Seeded weighted rollout: Empty secondary keys are dropped · case 05
Users with an empty secondary attribute bucket as if they had none.
Seeded weighted rollout: Sixteen hex digits overflow the unit interval · case 01
Nearly every user gets a bucket far above 1 and falls into the last variation.
Seeded weighted rollout: Sixteen hex digits overflow the unit interval · case 02
Nearly every user gets a bucket far above 1 and falls into the last variation.
Seeded weighted rollout: Sixteen hex digits overflow the unit interval · case 03
Nearly every user gets a bucket far above 1 and falls into the last variation.
Seeded weighted rollout: Sixteen hex digits overflow the unit interval · case 04
Nearly every user gets a bucket far above 1 and falls into the last variation.
Seeded weighted rollout: Sixteen hex digits overflow the unit interval · case 05
Nearly every user gets a bucket far above 1 and falls into the last variation.
Seeded weighted rollout: Buckets past the weights get no variation · case 01
When weights sum below 100000, some users evaluate to no variation at all.
Seeded weighted rollout: Buckets past the weights get no variation · case 02
When weights sum below 100000, some users evaluate to no variation at all.
Seeded weighted rollout: Buckets past the weights get no variation · case 03
When weights sum below 100000, some users evaluate to no variation at all.
Seeded weighted rollout: Buckets past the weights get no variation · case 04
When weights sum below 100000, some users evaluate to no variation at all.
Seeded weighted rollout: Buckets past the weights get no variation · case 05
When weights sum below 100000, some users evaluate to no variation at all.
Per-context evaluation memo: Memo ignores the flag version · case 01
After a flag is edited, clients keep serving results computed from the old version.
Per-context evaluation memo: Memo ignores the flag version · case 02
After a flag is edited, clients keep serving results computed from the old version.
Per-context evaluation memo: Memo ignores the flag version · case 03
After a flag is edited, clients keep serving results computed from the old version.
Per-context evaluation memo: Memo ignores the flag version · case 04
After a flag is edited, clients keep serving results computed from the old version.
Per-context evaluation memo: Memo ignores the flag version · case 05
After a flag is edited, clients keep serving results computed from the old version.
Per-context evaluation memo: Memo is keyed by context key only · case 01
A user who changes country keeps the result computed for the old country.
Per-context evaluation memo: Memo is keyed by context key only · case 02
A user who changes country keeps the result computed for the old country.
Per-context evaluation memo: Memo is keyed by context key only · case 03
A user who changes country keeps the result computed for the old country.
Per-context evaluation memo: Memo is keyed by context key only · case 04
A user who changes country keeps the result computed for the old country.
Per-context evaluation memo: Memo is keyed by context key only · case 05
A user who changes country keeps the result computed for the old country.
Per-context evaluation memo: Memo depends on attribute insertion order · case 01
Identical contexts built in different orders miss the cache and double evaluation counts.
Per-context evaluation memo: Memo depends on attribute insertion order · case 02
Identical contexts built in different orders miss the cache and double evaluation counts.
Per-context evaluation memo: Memo depends on attribute insertion order · case 03
Identical contexts built in different orders miss the cache and double evaluation counts.
Per-context evaluation memo: Memo depends on attribute insertion order · case 04
Identical contexts built in different orders miss the cache and double evaluation counts.
Per-context evaluation memo: Memo depends on attribute insertion order · case 05
Identical contexts built in different orders miss the cache and double evaluation counts.
Sample ratio mismatch check: Chi-square terms divide by the observed count · case 01
The SRM statistic is inflated for under-filled arms and deflated for over-filled ones.
Sample ratio mismatch check: Chi-square terms divide by the observed count · case 02
The SRM statistic is inflated for under-filled arms and deflated for over-filled ones.
Sample ratio mismatch check: Chi-square terms divide by the observed count · case 03
The SRM statistic is inflated for under-filled arms and deflated for over-filled ones.
Sample ratio mismatch check: Chi-square terms divide by the observed count · case 04
The SRM statistic is inflated for under-filled arms and deflated for over-filled ones.
Sample ratio mismatch check: Chi-square terms divide by the observed count · case 05
The SRM statistic is inflated for under-filled arms and deflated for over-filled ones.
Sample ratio mismatch check: Design weights are assumed to be equal · case 01
A deliberate 2:1 allocation is flagged as a sample ratio mismatch on every readout.
Sample ratio mismatch check: Design weights are assumed to be equal · case 02
A deliberate 2:1 allocation is flagged as a sample ratio mismatch on every readout.
Sample ratio mismatch check: Design weights are assumed to be equal · case 03
A deliberate 2:1 allocation is flagged as a sample ratio mismatch on every readout.
Sample ratio mismatch check: Design weights are assumed to be equal · case 04
A deliberate 2:1 allocation is flagged as a sample ratio mismatch on every readout.
Sample ratio mismatch check: Design weights are assumed to be equal · case 05
A deliberate 2:1 allocation is flagged as a sample ratio mismatch on every readout.
Sample ratio mismatch check: Zero-weight arms count toward degrees of freedom · case 01
Experiments with a drained arm use a looser critical value and miss real mismatches.
Sample ratio mismatch check: Zero-weight arms count toward degrees of freedom · case 02
Experiments with a drained arm use a looser critical value and miss real mismatches.
Sample ratio mismatch check: Zero-weight arms count toward degrees of freedom · case 03
Experiments with a drained arm use a looser critical value and miss real mismatches.
Sample ratio mismatch check: Zero-weight arms count toward degrees of freedom · case 04
Experiments with a drained arm use a looser critical value and miss real mismatches.
Sample ratio mismatch check: Zero-weight arms count toward degrees of freedom · case 05
Experiments with a drained arm use a looser critical value and miss real mismatches.
Sample ratio mismatch check: Units in a zero-weight arm are ignored · case 01
Users leaking into a disabled arm never trigger the mismatch alarm.
Sample ratio mismatch check: Units in a zero-weight arm are ignored · case 02
Users leaking into a disabled arm never trigger the mismatch alarm.
Sample ratio mismatch check: Units in a zero-weight arm are ignored · case 03
Users leaking into a disabled arm never trigger the mismatch alarm.
Sample ratio mismatch check: Units in a zero-weight arm are ignored · case 04
Users leaking into a disabled arm never trigger the mismatch alarm.
Sample ratio mismatch check: Units in a zero-weight arm are ignored · case 05
Users leaking into a disabled arm never trigger the mismatch alarm.
Sample ratio mismatch check: The mismatch threshold uses the 0.05 table · case 01
Healthy experiments are routinely flagged for SRM and readouts are blocked.
Sample ratio mismatch check: The mismatch threshold uses the 0.05 table · case 02
Healthy experiments are routinely flagged for SRM and readouts are blocked.
Sample ratio mismatch check: The mismatch threshold uses the 0.05 table · case 03
Healthy experiments are routinely flagged for SRM and readouts are blocked.
Sample ratio mismatch check: The mismatch threshold uses the 0.05 table · case 04
Healthy experiments are routinely flagged for SRM and readouts are blocked.
Sample ratio mismatch check: The mismatch threshold uses the 0.05 table · case 05
Healthy experiments are routinely flagged for SRM and readouts are blocked.
Conversion rate z-test: The pooled rate averages the two arm rates · case 01
With unequal arm sizes the null variance is wrong and p-values drift.
Conversion rate z-test: The pooled rate averages the two arm rates · case 02
With unequal arm sizes the null variance is wrong and p-values drift.
Conversion rate z-test: The pooled rate averages the two arm rates · case 03
With unequal arm sizes the null variance is wrong and p-values drift.
Conversion rate z-test: The pooled rate averages the two arm rates · case 04
With unequal arm sizes the null variance is wrong and p-values drift.
Conversion rate z-test: The pooled rate averages the two arm rates · case 05
With unequal arm sizes the null variance is wrong and p-values drift.
Conversion rate z-test: Standard error uses the combined sample size once · case 01
z-scores are roughly halved, so real effects look insignificant.
Conversion rate z-test: Standard error uses the combined sample size once · case 02
z-scores are roughly halved, so real effects look insignificant.
Conversion rate z-test: Standard error uses the combined sample size once · case 03
z-scores are roughly halved, so real effects look insignificant.
Conversion rate z-test: Standard error uses the combined sample size once · case 04
z-scores are roughly halved, so real effects look insignificant.
Conversion rate z-test: Standard error uses the combined sample size once · case 05
z-scores are roughly halved, so real effects look insignificant.
Conversion rate z-test: Two-sided p-values are halved · case 01
Twice as many experiments are declared significant as the alpha level allows.
Conversion rate z-test: Two-sided p-values are halved · case 02
Twice as many experiments are declared significant as the alpha level allows.
Conversion rate z-test: Two-sided p-values are halved · case 03
Twice as many experiments are declared significant as the alpha level allows.
Conversion rate z-test: Two-sided p-values are halved · case 04
Twice as many experiments are declared significant as the alpha level allows.
Conversion rate z-test: Two-sided p-values are halved · case 05
Twice as many experiments are declared significant as the alpha level allows.
Conversion rate z-test: The z statistic has the control-minus-treatment sign · case 01
Dashboards show winning treatments as losing.
Conversion rate z-test: The z statistic has the control-minus-treatment sign · case 02
Dashboards show winning treatments as losing.
Conversion rate z-test: The z statistic has the control-minus-treatment sign · case 03
Dashboards show winning treatments as losing.
Conversion rate z-test: The z statistic has the control-minus-treatment sign · case 04
Dashboards show winning treatments as losing.
Conversion rate z-test: The z statistic has the control-minus-treatment sign · case 05
Dashboards show winning treatments as losing.
Conversion rate z-test: Zero-variance data is reported as significant · case 01
Arms with no conversions at all are reported with p = 0.
Conversion rate z-test: Zero-variance data is reported as significant · case 02
Arms with no conversions at all are reported with p = 0.
Conversion rate z-test: Zero-variance data is reported as significant · case 03
Arms with no conversions at all are reported with p = 0.
Conversion rate z-test: Zero-variance data is reported as significant · case 04
Arms with no conversions at all are reported with p = 0.
Conversion rate z-test: Zero-variance data is reported as significant · case 05
Arms with no conversions at all are reported with p = 0.
Welch unequal-variance t statistic: Arm variances use the population divisor · case 01
Small arms get understated variance and overconfident t statistics.
Welch unequal-variance t statistic: Arm variances use the population divisor · case 02
Small arms get understated variance and overconfident t statistics.
Welch unequal-variance t statistic: Arm variances use the population divisor · case 03
Small arms get understated variance and overconfident t statistics.
Welch unequal-variance t statistic: Arm variances use the population divisor · case 04
Small arms get understated variance and overconfident t statistics.
Welch unequal-variance t statistic: Arm variances use the population divisor · case 05
Small arms get understated variance and overconfident t statistics.
Welch unequal-variance t statistic: Welch df divides by arm sizes · case 01
Degrees of freedom are inflated, making small-sample p-values too small.
Welch unequal-variance t statistic: Welch df divides by arm sizes · case 02
Degrees of freedom are inflated, making small-sample p-values too small.
Welch unequal-variance t statistic: Welch df divides by arm sizes · case 03
Degrees of freedom are inflated, making small-sample p-values too small.
Welch unequal-variance t statistic: Welch df divides by arm sizes · case 04
Degrees of freedom are inflated, making small-sample p-values too small.
Welch unequal-variance t statistic: Welch df divides by arm sizes · case 05
Degrees of freedom are inflated, making small-sample p-values too small.
INSPECTABLE BY DESIGN
Every result has a runnable source.
Runnable implementations with recorded outputs, source hashes, and explicit contracts. Related variants share a failure mechanism and belong together in evaluation splits.
Read the methodology ↗