Understand the failure.
Verify the repair.
Small, reproducible software failures. The broken implementation, the fix that didn’t work, and the one that passed—preserved together.
Explore the cases ↓How results are verified ↗WHAT THE ARCHIVE CONTAINS
100840 executable cases. 20168 are open.
Every case records the implementation that fails, the fix that did not work, and the repair that passed its checks—with recorded outputs and source hashes. This release adds 100840 cases across 20168 failure mechanisms and 254 domains.
The open tier gives you the failure and the unsuccessful fix for one case in every mechanism. The remaining 80672 cases, 5 variants per mechanism, are member-only: the verified repair, its recorded checks, and the full fixture suite are held in the member archive. Read the methodology ↗
A RECORD OF WHAT WENT WRONG
Browse the archive / 100840
Pooling a disproportionate stratified sample changes expansion totals · case 01
Pooling a disproportionate stratified sample changes expansion totals.
Pooling a disproportionate stratified sample changes expansion totals · case 02
Pooling a disproportionate stratified sample changes expansion totals.
Pooling a disproportionate stratified sample changes expansion totals · case 03
Pooling a disproportionate stratified sample changes expansion totals.
Pooling a disproportionate stratified sample changes expansion totals · case 04
Pooling a disproportionate stratified sample changes expansion totals.
Pooling a disproportionate stratified sample changes expansion totals · case 05
Pooling a disproportionate stratified sample changes expansion totals.
Domain filtering silently changes the target denominator · case 01
Domain filtering silently changes the target denominator.
Domain filtering silently changes the target denominator · case 02
Domain filtering silently changes the target denominator.
Domain filtering silently changes the target denominator · case 03
Domain filtering silently changes the target denominator.
Domain filtering silently changes the target denominator · case 04
Domain filtering silently changes the target denominator.
Domain filtering silently changes the target denominator · case 05
Domain filtering silently changes the target denominator.
A global response adjustment transfers weight across response cells · case 01
A global response adjustment transfers weight across response cells.
A global response adjustment transfers weight across response cells · case 02
A global response adjustment transfers weight across response cells.
A global response adjustment transfers weight across response cells · case 03
A global response adjustment transfers weight across response cells.
A global response adjustment transfers weight across response cells · case 04
A global response adjustment transfers weight across response cells.
A global response adjustment transfers weight across response cells · case 05
A global response adjustment transfers weight across response cells.
Calibration matches record counts rather than population margins · case 01
Calibration matches record counts rather than population margins.
Calibration matches record counts rather than population margins · case 02
Calibration matches record counts rather than population margins.
Calibration matches record counts rather than population margins · case 03
Calibration matches record counts rather than population margins.
Calibration matches record counts rather than population margins · case 04
Calibration matches record counts rather than population margins.
Calibration matches record counts rather than population margins · case 05
Calibration matches record counts rather than population margins.
A census retains sampling variance · case 01
A census retains sampling variance.
A census retains sampling variance · case 02
A census retains sampling variance.
A census retains sampling variance · case 03
A census retains sampling variance.
A census retains sampling variance · case 04
A census retains sampling variance.
A census retains sampling variance · case 05
A census retains sampling variance.
Secondary records inflate survey variance degrees of freedom · case 01
Secondary records inflate survey variance degrees of freedom.
Secondary records inflate survey variance degrees of freedom · case 02
Secondary records inflate survey variance degrees of freedom.
Secondary records inflate survey variance degrees of freedom · case 03
Secondary records inflate survey variance degrees of freedom.
Secondary records inflate survey variance degrees of freedom · case 04
Secondary records inflate survey variance degrees of freedom.
Secondary records inflate survey variance degrees of freedom · case 05
Secondary records inflate survey variance degrees of freedom.
Fay replicate shrinkage understates variance · case 01
Fay replicate shrinkage understates variance.
Fay replicate shrinkage understates variance · case 02
Fay replicate shrinkage understates variance.
Fay replicate shrinkage understates variance · case 03
Fay replicate shrinkage understates variance.
Fay replicate shrinkage understates variance · case 04
Fay replicate shrinkage understates variance.
Fay replicate shrinkage understates variance · case 05
Fay replicate shrinkage understates variance.
Deduplicating repeated PPS draws changes the estimator · case 01
Deduplicating repeated PPS draws changes the estimator.
Deduplicating repeated PPS draws changes the estimator · case 02
Deduplicating repeated PPS draws changes the estimator.
Deduplicating repeated PPS draws changes the estimator · case 03
Deduplicating repeated PPS draws changes the estimator.
Deduplicating repeated PPS draws changes the estimator · case 04
Deduplicating repeated PPS draws changes the estimator.
Deduplicating repeated PPS draws changes the estimator · case 05
Deduplicating repeated PPS draws changes the estimator.
Intention-to-treat groups are rebuilt from received treatment · case 01
Intention-to-treat groups are rebuilt from received treatment.
Intention-to-treat groups are rebuilt from received treatment · case 02
Intention-to-treat groups are rebuilt from received treatment.
Intention-to-treat groups are rebuilt from received treatment · case 03
Intention-to-treat groups are rebuilt from received treatment.
Intention-to-treat groups are rebuilt from received treatment · case 04
Intention-to-treat groups are rebuilt from received treatment.
Intention-to-treat groups are rebuilt from received treatment · case 05
Intention-to-treat groups are rebuilt from received treatment.
An ATT request silently averages untreated potential outcomes · case 01
An ATT request silently averages untreated potential outcomes.
An ATT request silently averages untreated potential outcomes · case 02
An ATT request silently averages untreated potential outcomes.
An ATT request silently averages untreated potential outcomes · case 03
An ATT request silently averages untreated potential outcomes.
An ATT request silently averages untreated potential outcomes · case 04
An ATT request silently averages untreated potential outcomes.
An ATT request silently averages untreated potential outcomes · case 05
An ATT request silently averages untreated potential outcomes.
Post-assignment covariates enter a prespecified baseline adjustment set · case 01
Post-assignment covariates enter a prespecified baseline adjustment set.
Post-assignment covariates enter a prespecified baseline adjustment set · case 02
Post-assignment covariates enter a prespecified baseline adjustment set.
Post-assignment covariates enter a prespecified baseline adjustment set · case 03
Post-assignment covariates enter a prespecified baseline adjustment set.
Post-assignment covariates enter a prespecified baseline adjustment set · case 04
Post-assignment covariates enter a prespecified baseline adjustment set.
Post-assignment covariates enter a prespecified baseline adjustment set · case 05
Post-assignment covariates enter a prespecified baseline adjustment set.
Global treatment overlap conceals an unsupported target stratum · case 01
Global treatment overlap conceals an unsupported target stratum.
Global treatment overlap conceals an unsupported target stratum · case 02
Global treatment overlap conceals an unsupported target stratum.
Global treatment overlap conceals an unsupported target stratum · case 03
Global treatment overlap conceals an unsupported target stratum.
Global treatment overlap conceals an unsupported target stratum · case 04
Global treatment overlap conceals an unsupported target stratum.
Global treatment overlap conceals an unsupported target stratum · case 05
Global treatment overlap conceals an unsupported target stratum.
Cluster randomization accepts contradictory unit assignments · case 01
Cluster randomization accepts contradictory unit assignments.
Cluster randomization accepts contradictory unit assignments · case 02
Cluster randomization accepts contradictory unit assignments.
Cluster randomization accepts contradictory unit assignments · case 03
Cluster randomization accepts contradictory unit assignments.
Cluster randomization accepts contradictory unit assignments · case 04
Cluster randomization accepts contradictory unit assignments.
Cluster randomization accepts contradictory unit assignments · case 05
Cluster randomization accepts contradictory unit assignments.
Exposure mapping includes the focal unit as its own peer · case 01
Exposure mapping includes the focal unit as its own peer.
Exposure mapping includes the focal unit as its own peer · case 02
Exposure mapping includes the focal unit as its own peer.
Exposure mapping includes the focal unit as its own peer · case 03
Exposure mapping includes the focal unit as its own peer.
Exposure mapping includes the focal unit as its own peer · case 04
Exposure mapping includes the focal unit as its own peer.
Exposure mapping includes the focal unit as its own peer · case 05
Exposure mapping includes the focal unit as its own peer.
A synthetic Wald ratio divides by uptake level instead of uptake contrast · case 01
A synthetic Wald ratio divides by uptake level instead of uptake contrast.
A synthetic Wald ratio divides by uptake level instead of uptake contrast · case 02
A synthetic Wald ratio divides by uptake level instead of uptake contrast.
A synthetic Wald ratio divides by uptake level instead of uptake contrast · case 03
A synthetic Wald ratio divides by uptake level instead of uptake contrast.
A synthetic Wald ratio divides by uptake level instead of uptake contrast · case 04
A synthetic Wald ratio divides by uptake level instead of uptake contrast.
A synthetic Wald ratio divides by uptake level instead of uptake contrast · case 05
A synthetic Wald ratio divides by uptake level instead of uptake contrast.
A two-period design subtracts levels instead of changes · case 01
A two-period design subtracts levels instead of changes.
A two-period design subtracts levels instead of changes · case 02
A two-period design subtracts levels instead of changes.
A two-period design subtracts levels instead of changes · case 03
A two-period design subtracts levels instead of changes.
A two-period design subtracts levels instead of changes · case 04
A two-period design subtracts levels instead of changes.
A two-period design subtracts levels instead of changes · case 05
A two-period design subtracts levels instead of changes.
Crossover carryover coding forgets actual prior exposure across untreated periods · case 01
A carryover covariate follows assigned treatment or disappears during an untreated period without a documented washout.
Crossover carryover coding forgets actual prior exposure across untreated periods · case 02
A carryover covariate follows assigned treatment or disappears during an untreated period without a documented washout.
Crossover carryover coding forgets actual prior exposure across untreated periods · case 03
A carryover covariate follows assigned treatment or disappears during an untreated period without a documented washout.
Crossover carryover coding forgets actual prior exposure across untreated periods · case 04
A carryover covariate follows assigned treatment or disappears during an untreated period without a documented washout.
Crossover carryover coding forgets actual prior exposure across untreated periods · case 05
A carryover covariate follows assigned treatment or disappears during an untreated period without a documented washout.
Ever-treated labeling allocates pre-initiation time to treatment · case 01
Ever-treated labeling allocates pre-initiation time to treatment.
Ever-treated labeling allocates pre-initiation time to treatment · case 02
Ever-treated labeling allocates pre-initiation time to treatment.
Ever-treated labeling allocates pre-initiation time to treatment · case 03
Ever-treated labeling allocates pre-initiation time to treatment.
Ever-treated labeling allocates pre-initiation time to treatment · case 04
Ever-treated labeling allocates pre-initiation time to treatment.
Ever-treated labeling allocates pre-initiation time to treatment · case 05
Ever-treated labeling allocates pre-initiation time to treatment.
A reserve-qualified winner pays its own bid · case 01
A reserve-qualified winner pays its own bid.
A reserve-qualified winner pays its own bid · case 02
A reserve-qualified winner pays its own bid.
A reserve-qualified winner pays its own bid · case 03
A reserve-qualified winner pays its own bid.
A reserve-qualified winner pays its own bid · case 04
A reserve-qualified winner pays its own bid.
A reserve-qualified winner pays its own bid · case 05
A reserve-qualified winner pays its own bid.
Uniform unit pricing reads the last winner instead of the first loser · case 01
Uniform unit pricing reads the last winner instead of the first loser.
Uniform unit pricing reads the last winner instead of the first loser · case 02
Uniform unit pricing reads the last winner instead of the first loser.
Uniform unit pricing reads the last winner instead of the first loser · case 03
Uniform unit pricing reads the last winner instead of the first loser.
Uniform unit pricing reads the last winner instead of the first loser · case 04
Uniform unit pricing reads the last winner instead of the first loser.
Uniform unit pricing reads the last winner instead of the first loser · case 05
Uniform unit pricing reads the last winner instead of the first loser.
INSPECTABLE BY DESIGN
Every result has a runnable source.
Runnable implementations with recorded outputs, source hashes, and explicit contracts. Related variants share a failure mechanism and belong together in evaluation splits.
Read the methodology ↗