Understand the failure.
Verify the repair.
Small, reproducible software failures. The broken implementation, the fix that didn’t work, and the one that passed—preserved together.
Explore the cases ↓How results are verified ↗WHAT THE ARCHIVE CONTAINS
100840 executable cases. 20168 are open.
Every case records the implementation that fails, the fix that did not work, and the repair that passed its checks—with recorded outputs and source hashes. This release adds 100840 cases across 20168 failure mechanisms and 254 domains.
The open tier gives you the failure and the unsuccessful fix for one case in every mechanism. The remaining 80672 cases, 5 variants per mechanism, are member-only: the verified repair, its recorded checks, and the full fixture suite are held in the member archive. Read the methodology ↗
A RECORD OF WHAT WENT WRONG
Browse the archive / 100840
Multilabel micro recall counts samples instead of labels · case 01
Multilabel micro recall counts samples instead of labels.
Multilabel micro recall counts samples instead of labels · case 02
Multilabel micro recall counts samples instead of labels.
Multilabel micro recall counts samples instead of labels · case 03
Multilabel micro recall counts samples instead of labels.
Multilabel micro recall counts samples instead of labels · case 04
Multilabel micro recall counts samples instead of labels.
Multilabel micro recall counts samples instead of labels · case 05
Multilabel micro recall counts samples instead of labels.
Ignored labels remain in the accuracy denominator · case 01
Ignored labels remain in the accuracy denominator.
Ignored labels remain in the accuracy denominator · case 02
Ignored labels remain in the accuracy denominator.
Ignored labels remain in the accuracy denominator · case 03
Ignored labels remain in the accuracy denominator.
Ignored labels remain in the accuracy denominator · case 04
Ignored labels remain in the accuracy denominator.
Ignored labels remain in the accuracy denominator · case 05
Ignored labels remain in the accuracy denominator.
Sample weights are applied only to correct rows · case 01
Sample weights are applied only to correct rows.
Sample weights are applied only to correct rows · case 02
Sample weights are applied only to correct rows.
Sample weights are applied only to correct rows · case 03
Sample weights are applied only to correct rows.
Sample weights are applied only to correct rows · case 04
Sample weights are applied only to correct rows.
Sample weights are applied only to correct rows · case 05
Sample weights are applied only to correct rows.
Threshold sweep reuses the previous threshold counts · case 01
Threshold sweep reuses the previous threshold counts.
Threshold sweep reuses the previous threshold counts · case 02
Threshold sweep reuses the previous threshold counts.
Threshold sweep reuses the previous threshold counts · case 03
Threshold sweep reuses the previous threshold counts.
Threshold sweep reuses the previous threshold counts · case 04
Threshold sweep reuses the previous threshold counts.
Threshold sweep reuses the previous threshold counts · case 05
Threshold sweep reuses the previous threshold counts.
Grouped accuracy includes empty groups as zero · case 01
Grouped accuracy includes empty groups as zero.
Grouped accuracy includes empty groups as zero · case 02
Grouped accuracy includes empty groups as zero.
Grouped accuracy includes empty groups as zero · case 03
Grouped accuracy includes empty groups as zero.
Grouped accuracy includes empty groups as zero · case 04
Grouped accuracy includes empty groups as zero.
Grouped accuracy includes empty groups as zero · case 05
Grouped accuracy includes empty groups as zero.
Row-normalized confusion cells use column totals · case 01
Row-normalized confusion cells use column totals.
Row-normalized confusion cells use column totals · case 02
Row-normalized confusion cells use column totals.
Row-normalized confusion cells use column totals · case 03
Row-normalized confusion cells use column totals.
Row-normalized confusion cells use column totals · case 04
Row-normalized confusion cells use column totals.
Row-normalized confusion cells use column totals · case 05
Row-normalized confusion cells use column totals.
Multiclass MCC reports accuracy instead of correlation · case 01
Multiclass MCC reports accuracy instead of correlation.
Multiclass MCC reports accuracy instead of correlation · case 02
Multiclass MCC reports accuracy instead of correlation.
Multiclass MCC reports accuracy instead of correlation · case 03
Multiclass MCC reports accuracy instead of correlation.
Multiclass MCC reports accuracy instead of correlation · case 04
Multiclass MCC reports accuracy instead of correlation.
Multiclass MCC reports accuracy instead of correlation · case 05
Multiclass MCC reports accuracy instead of correlation.
Slice aggregation gives each nonempty slice equal weight · case 01
Slice aggregation gives each nonempty slice equal weight.
Slice aggregation gives each nonempty slice equal weight · case 02
Slice aggregation gives each nonempty slice equal weight.
Slice aggregation gives each nonempty slice equal weight · case 03
Slice aggregation gives each nonempty slice equal weight.
Slice aggregation gives each nonempty slice equal weight · case 04
Slice aggregation gives each nonempty slice equal weight.
Slice aggregation gives each nonempty slice equal weight · case 05
Slice aggregation gives each nonempty slice equal weight.
Preference pairs: responses are paired by list position · case 01
responses are paired by list position.
Preference pairs: responses are paired by list position · case 02
responses are paired by list position.
Preference pairs: responses are paired by list position · case 03
responses are paired by list position.
Preference pairs: responses are paired by list position · case 04
responses are paired by list position.
Preference pairs: responses are paired by list position · case 05
responses are paired by list position.
Preference pairs: equal ratings produce an oriented pair · case 01
equal ratings produce an oriented pair.
Preference pairs: equal ratings produce an oriented pair · case 02
equal ratings produce an oriented pair.
Preference pairs: equal ratings produce an oriented pair · case 03
equal ratings produce an oriented pair.
Preference pairs: equal ratings produce an oriented pair · case 04
equal ratings produce an oriented pair.
Preference pairs: equal ratings produce an oriented pair · case 05
equal ratings produce an oriented pair.
Preference pairs: ineligible responses become pair endpoints · case 01
ineligible responses become pair endpoints.
Preference pairs: ineligible responses become pair endpoints · case 02
ineligible responses become pair endpoints.
Preference pairs: ineligible responses become pair endpoints · case 03
ineligible responses become pair endpoints.
Preference pairs: ineligible responses become pair endpoints · case 04
ineligible responses become pair endpoints.
Preference pairs: ineligible responses become pair endpoints · case 05
ineligible responses become pair endpoints.
Preference pairs: responses from different prompts are compared · case 01
responses from different prompts are compared.
Preference pairs: responses from different prompts are compared · case 02
responses from different prompts are compared.
Preference pairs: responses from different prompts are compared · case 03
responses from different prompts are compared.
Preference pairs: responses from different prompts are compared · case 04
responses from different prompts are compared.
Preference pairs: responses from different prompts are compared · case 05
responses from different prompts are compared.
Preference pairs: replayed response IDs create duplicate pairs · case 01
replayed response IDs create duplicate pairs.
Preference pairs: replayed response IDs create duplicate pairs · case 02
replayed response IDs create duplicate pairs.
Preference pairs: replayed response IDs create duplicate pairs · case 03
replayed response IDs create duplicate pairs.
Preference pairs: replayed response IDs create duplicate pairs · case 04
replayed response IDs create duplicate pairs.
Preference pairs: replayed response IDs create duplicate pairs · case 05
replayed response IDs create duplicate pairs.
Preference pairs: parallel candidate arrays drift out of alignment · case 01
parallel candidate arrays drift out of alignment.
Preference pairs: parallel candidate arrays drift out of alignment · case 02
parallel candidate arrays drift out of alignment.
Preference pairs: parallel candidate arrays drift out of alignment · case 03
parallel candidate arrays drift out of alignment.
Preference pairs: parallel candidate arrays drift out of alignment · case 04
parallel candidate arrays drift out of alignment.
Preference pairs: parallel candidate arrays drift out of alignment · case 05
parallel candidate arrays drift out of alignment.
Preference pairs: sub-threshold rating gaps are accepted · case 01
sub-threshold rating gaps are accepted.
Preference pairs: sub-threshold rating gaps are accepted · case 02
sub-threshold rating gaps are accepted.
Preference pairs: sub-threshold rating gaps are accepted · case 03
sub-threshold rating gaps are accepted.
Preference pairs: sub-threshold rating gaps are accepted · case 04
sub-threshold rating gaps are accepted.
Preference pairs: sub-threshold rating gaps are accepted · case 05
sub-threshold rating gaps are accepted.
Preference pairs: pair weights remain unnormalized · case 01
pair weights remain unnormalized.
Preference pairs: pair weights remain unnormalized · case 02
pair weights remain unnormalized.
Preference pairs: pair weights remain unnormalized · case 03
pair weights remain unnormalized.
Preference pairs: pair weights remain unnormalized · case 04
pair weights remain unnormalized.
Preference pairs: pair weights remain unnormalized · case 05
pair weights remain unnormalized.
Preference pairs: weights are normalized across unrelated prompts · case 01
weights are normalized across unrelated prompts.
Preference pairs: weights are normalized across unrelated prompts · case 02
weights are normalized across unrelated prompts.
Preference pairs: weights are normalized across unrelated prompts · case 03
weights are normalized across unrelated prompts.
Preference pairs: weights are normalized across unrelated prompts · case 04
weights are normalized across unrelated prompts.
Preference pairs: weights are normalized across unrelated prompts · case 05
weights are normalized across unrelated prompts.
Preference pairs: only adjacent candidates are paired · case 01
only adjacent candidates are paired.
Preference pairs: only adjacent candidates are paired · case 02
only adjacent candidates are paired.
Preference pairs: only adjacent candidates are paired · case 03
only adjacent candidates are paired.
Preference pairs: only adjacent candidates are paired · case 04
only adjacent candidates are paired.
Preference pairs: only adjacent candidates are paired · case 05
only adjacent candidates are paired.
Preference pairs: tied extrema are resolved by input order · case 01
tied extrema are resolved by input order.
Preference pairs: tied extrema are resolved by input order · case 02
tied extrema are resolved by input order.
Preference pairs: tied extrema are resolved by input order · case 03
tied extrema are resolved by input order.
Preference pairs: tied extrema are resolved by input order · case 04
tied extrema are resolved by input order.
Preference pairs: tied extrema are resolved by input order · case 05
tied extrema are resolved by input order.
Preference pairs: pair cap is applied before duplicate removal · case 01
pair cap is applied before duplicate removal.
Preference pairs: pair cap is applied before duplicate removal · case 02
pair cap is applied before duplicate removal.
Preference pairs: pair cap is applied before duplicate removal · case 03
pair cap is applied before duplicate removal.
Preference pairs: pair cap is applied before duplicate removal · case 04
pair cap is applied before duplicate removal.
Preference pairs: pair cap is applied before duplicate removal · case 05
pair cap is applied before duplicate removal.
INSPECTABLE BY DESIGN
Every result has a runnable source.
Runnable implementations with recorded outputs, source hashes, and explicit contracts. Related variants share a failure mechanism and belong together in evaluation splits.
Read the methodology ↗