Understand the failure.
Verify the repair.
Small, reproducible software failures. The broken implementation, the fix that didn’t work, and the one that passed—preserved together.
Explore the cases ↓How results are verified ↗WHAT THE ARCHIVE CONTAINS
100840 executable cases. 20168 are open.
Every case records the implementation that fails, the fix that did not work, and the repair that passed its checks—with recorded outputs and source hashes. This release adds 100840 cases across 20168 failure mechanisms and 254 domains.
The open tier gives you the failure and the unsuccessful fix for one case in every mechanism. The remaining 80672 cases, 5 variants per mechanism, are member-only: the verified repair, its recorded checks, and the full fixture suite are held in the member archive. Read the methodology ↗
A RECORD OF WHAT WENT WRONG
Browse the archive / 100840
Batch sampler selection is ignored when labels are paired · case 01
Batch sampler selection is ignored when labels are paired.
Batch sampler selection is ignored when labels are paired · case 02
Batch sampler selection is ignored when labels are paired.
Batch sampler selection is ignored when labels are paired · case 03
Batch sampler selection is ignored when labels are paired.
Batch sampler selection is ignored when labels are paired · case 04
Batch sampler selection is ignored when labels are paired.
Batch sampler selection is ignored when labels are paired · case 05
Batch sampler selection is ignored when labels are paired.
Vocabulary construction counts repeated labels as separate classes · case 01
Vocabulary construction counts repeated labels as separate classes.
Vocabulary construction counts repeated labels as separate classes · case 02
Vocabulary construction counts repeated labels as separate classes.
Vocabulary construction counts repeated labels as separate classes · case 03
Vocabulary construction counts repeated labels as separate classes.
Vocabulary construction counts repeated labels as separate classes · case 04
Vocabulary construction counts repeated labels as separate classes.
Vocabulary construction counts repeated labels as separate classes · case 05
Vocabulary construction counts repeated labels as separate classes.
Student targets use teacher class indices directly · case 01
Student targets use teacher class indices directly.
Student targets use teacher class indices directly · case 02
Student targets use teacher class indices directly.
Student targets use teacher class indices directly · case 03
Student targets use teacher class indices directly.
Student targets use teacher class indices directly · case 04
Student targets use teacher class indices directly.
Student targets use teacher class indices directly · case 05
Student targets use teacher class indices directly.
Sequence-level labels are taken from the first token · case 01
Sequence-level labels are taken from the first token.
Sequence-level labels are taken from the first token · case 02
Sequence-level labels are taken from the first token.
Sequence-level labels are taken from the first token · case 03
Sequence-level labels are taken from the first token.
Sequence-level labels are taken from the first token · case 04
Sequence-level labels are taken from the first token.
Sequence-level labels are taken from the first token · case 05
Sequence-level labels are taken from the first token.
Augmented copies receive the base weight instead of divided weight · case 01
Augmented copies receive the base weight instead of divided weight.
Augmented copies receive the base weight instead of divided weight · case 02
Augmented copies receive the base weight instead of divided weight.
Augmented copies receive the base weight instead of divided weight · case 03
Augmented copies receive the base weight instead of divided weight.
Augmented copies receive the base weight instead of divided weight · case 04
Augmented copies receive the base weight instead of divided weight.
Augmented copies receive the base weight instead of divided weight · case 05
Augmented copies receive the base weight instead of divided weight.
Ordinal labels use a single class bit instead of cumulative thresholds · case 01
Ordinal labels use a single class bit instead of cumulative thresholds.
Ordinal labels use a single class bit instead of cumulative thresholds · case 02
Ordinal labels use a single class bit instead of cumulative thresholds.
Ordinal labels use a single class bit instead of cumulative thresholds · case 03
Ordinal labels use a single class bit instead of cumulative thresholds.
Ordinal labels use a single class bit instead of cumulative thresholds · case 04
Ordinal labels use a single class bit instead of cumulative thresholds.
Ordinal labels use a single class bit instead of cumulative thresholds · case 05
Ordinal labels use a single class bit instead of cumulative thresholds.
Smoothed targets depend on classes observed in the batch · case 01
Smoothed targets depend on classes observed in the batch.
Smoothed targets depend on classes observed in the batch · case 02
Smoothed targets depend on classes observed in the batch.
Smoothed targets depend on classes observed in the batch · case 03
Smoothed targets depend on classes observed in the batch.
Smoothed targets depend on classes observed in the batch · case 04
Smoothed targets depend on classes observed in the batch.
Smoothed targets depend on classes observed in the batch · case 05
Smoothed targets depend on classes observed in the batch.
Balanced example weights divide by total examples instead of class frequency · case 01
Balanced example weights divide by total examples instead of class frequency.
Balanced example weights divide by total examples instead of class frequency · case 02
Balanced example weights divide by total examples instead of class frequency.
Balanced example weights divide by total examples instead of class frequency · case 03
Balanced example weights divide by total examples instead of class frequency.
Balanced example weights divide by total examples instead of class frequency · case 04
Balanced example weights divide by total examples instead of class frequency.
Balanced example weights divide by total examples instead of class frequency · case 05
Balanced example weights divide by total examples instead of class frequency.
Greedy packing flushes one token too early · case 01
Greedy packing flushes one token too early.
Greedy packing flushes one token too early · case 02
Greedy packing flushes one token too early.
Greedy packing flushes one token too early · case 03
Greedy packing flushes one token too early.
Greedy packing flushes one token too early · case 04
Greedy packing flushes one token too early.
Greedy packing flushes one token too early · case 05
Greedy packing flushes one token too early.
Oversized sequences are split into artificial chunks · case 01
Oversized sequences are split into artificial chunks.
Oversized sequences are split into artificial chunks · case 02
Oversized sequences are split into artificial chunks.
Oversized sequences are split into artificial chunks · case 03
Oversized sequences are split into artificial chunks.
Oversized sequences are split into artificial chunks · case 04
Oversized sequences are split into artificial chunks.
Oversized sequences are split into artificial chunks · case 05
Oversized sequences are split into artificial chunks.
Batch padding rounds down to a length multiple · case 01
Batch padding rounds down to a length multiple.
Batch padding rounds down to a length multiple · case 02
Batch padding rounds down to a length multiple.
Batch padding rounds down to a length multiple · case 03
Batch padding rounds down to a length multiple.
Batch padding rounds down to a length multiple · case 04
Batch padding rounds down to a length multiple.
Batch padding rounds down to a length multiple · case 05
Batch padding rounds down to a length multiple.
Left padding marks padded prefix tokens as valid · case 01
Left padding marks padded prefix tokens as valid.
Left padding marks padded prefix tokens as valid · case 02
Left padding marks padded prefix tokens as valid.
Left padding marks padded prefix tokens as valid · case 03
Left padding marks padded prefix tokens as valid.
Left padding marks padded prefix tokens as valid · case 04
Left padding marks padded prefix tokens as valid.
Left padding marks padded prefix tokens as valid · case 05
Left padding marks padded prefix tokens as valid.
Right padding mask includes the first padded position · case 01
Right padding mask includes the first padded position.
Right padding mask includes the first padded position · case 02
Right padding mask includes the first padded position.
Right padding mask includes the first padded position · case 03
Right padding mask includes the first padded position.
Right padding mask includes the first padded position · case 04
Right padding mask includes the first padded position.
Right padding mask includes the first padded position · case 05
Right padding mask includes the first padded position.
Separator tokens are omitted from the packing budget · case 01
Separator tokens are omitted from the packing budget.
Separator tokens are omitted from the packing budget · case 02
Separator tokens are omitted from the packing budget.
Separator tokens are omitted from the packing budget · case 03
Separator tokens are omitted from the packing budget.
Separator tokens are omitted from the packing budget · case 04
Separator tokens are omitted from the packing budget.
Separator tokens are omitted from the packing budget · case 05
Separator tokens are omitted from the packing budget.
Fixed-width chunks use floor instead of ceiling count · case 01
Fixed-width chunks use floor instead of ceiling count.
Fixed-width chunks use floor instead of ceiling count · case 02
Fixed-width chunks use floor instead of ceiling count.
Fixed-width chunks use floor instead of ceiling count · case 03
Fixed-width chunks use floor instead of ceiling count.
Fixed-width chunks use floor instead of ceiling count · case 04
Fixed-width chunks use floor instead of ceiling count.
Fixed-width chunks use floor instead of ceiling count · case 05
Fixed-width chunks use floor instead of ceiling count.
Overlapping windows advance by the overlap instead of stride · case 01
Overlapping windows advance by the overlap instead of stride.
Overlapping windows advance by the overlap instead of stride · case 02
Overlapping windows advance by the overlap instead of stride.
Overlapping windows advance by the overlap instead of stride · case 03
Overlapping windows advance by the overlap instead of stride.
Overlapping windows advance by the overlap instead of stride · case 04
Overlapping windows advance by the overlap instead of stride.
Overlapping windows advance by the overlap instead of stride · case 05
Overlapping windows advance by the overlap instead of stride.
Segment identifiers continue across packed rows · case 01
Segment identifiers continue across packed rows.
Segment identifiers continue across packed rows · case 02
Segment identifiers continue across packed rows.
Segment identifiers continue across packed rows · case 03
Segment identifiers continue across packed rows.
Segment identifiers continue across packed rows · case 04
Segment identifiers continue across packed rows.
Segment identifiers continue across packed rows · case 05
Segment identifiers continue across packed rows.
Packed positions count across segment boundaries · case 01
Packed positions count across segment boundaries.
Packed positions count across segment boundaries · case 02
Packed positions count across segment boundaries.
Packed positions count across segment boundaries · case 03
Packed positions count across segment boundaries.
Packed positions count across segment boundaries · case 04
Packed positions count across segment boundaries.
Packed positions count across segment boundaries · case 05
Packed positions count across segment boundaries.
Rows exceed the configured sequence count · case 01
Rows exceed the configured sequence count.
Rows exceed the configured sequence count · case 02
Rows exceed the configured sequence count.
Rows exceed the configured sequence count · case 03
Rows exceed the configured sequence count.
Rows exceed the configured sequence count · case 04
Rows exceed the configured sequence count.
Rows exceed the configured sequence count · case 05
Rows exceed the configured sequence count.
Length bucketing reverses equal-length input order · case 01
Length bucketing reverses equal-length input order.
Length bucketing reverses equal-length input order · case 02
Length bucketing reverses equal-length input order.
Length bucketing reverses equal-length input order · case 03
Length bucketing reverses equal-length input order.
Length bucketing reverses equal-length input order · case 04
Length bucketing reverses equal-length input order.
Length bucketing reverses equal-length input order · case 05
Length bucketing reverses equal-length input order.
INSPECTABLE BY DESIGN
Every result has a runnable source.
Runnable implementations with recorded outputs, source hashes, and explicit contracts. Related variants share a failure mechanism and belong together in evaluation splits.
Read the methodology ↗