Understand the failure.
Verify the repair.
Small, reproducible software failures. The broken implementation, the fix that didn’t work, and the one that passed—preserved together.
Explore the cases ↓How results are verified ↗WHAT THE ARCHIVE CONTAINS
100840 executable cases. 20168 are open.
Every case records the implementation that fails, the fix that did not work, and the repair that passed its checks—with recorded outputs and source hashes. This release adds 100840 cases across 20168 failure mechanisms and 254 domains.
The open tier gives you the failure and the unsuccessful fix for one case in every mechanism. The remaining 80672 cases, 5 variants per mechanism, are member-only: the verified repair, its recorded checks, and the full fixture suite are held in the member archive. Read the methodology ↗
A RECORD OF WHAT WENT WRONG
Browse the archive / 100840
Length bin packing uses next-fit instead of first-fit · case 01
Length bin packing uses next-fit instead of first-fit.
Length bin packing uses next-fit instead of first-fit · case 02
Length bin packing uses next-fit instead of first-fit.
Length bin packing uses next-fit instead of first-fit · case 03
Length bin packing uses next-fit instead of first-fit.
Length bin packing uses next-fit instead of first-fit · case 04
Length bin packing uses next-fit instead of first-fit.
Length bin packing uses next-fit instead of first-fit · case 05
Length bin packing uses next-fit instead of first-fit.
Round-robin packing consumes one source unevenly · case 01
Round-robin packing consumes one source unevenly.
Round-robin packing consumes one source unevenly · case 02
Round-robin packing consumes one source unevenly.
Round-robin packing consumes one source unevenly · case 03
Round-robin packing consumes one source unevenly.
Round-robin packing consumes one source unevenly · case 04
Round-robin packing consumes one source unevenly.
Round-robin packing consumes one source unevenly · case 05
Round-robin packing consumes one source unevenly.
Segment spans include the next segment boundary · case 01
Segment spans include the next segment boundary.
Segment spans include the next segment boundary · case 02
Segment spans include the next segment boundary.
Segment spans include the next segment boundary · case 03
Segment spans include the next segment boundary.
Segment spans include the next segment boundary · case 04
Segment spans include the next segment boundary.
Segment spans include the next segment boundary · case 05
Segment spans include the next segment boundary.
Continuation flags mislabel chunk boundaries · case 01
Continuation flags mislabel chunk boundaries.
Continuation flags mislabel chunk boundaries · case 02
Continuation flags mislabel chunk boundaries.
Continuation flags mislabel chunk boundaries · case 03
Continuation flags mislabel chunk boundaries.
Continuation flags mislabel chunk boundaries · case 04
Continuation flags mislabel chunk boundaries.
Continuation flags mislabel chunk boundaries · case 05
Continuation flags mislabel chunk boundaries.
Loss mask includes prompt tokens · case 01
Loss mask includes prompt tokens.
Loss mask includes prompt tokens · case 02
Loss mask includes prompt tokens.
Loss mask includes prompt tokens · case 03
Loss mask includes prompt tokens.
Loss mask includes prompt tokens · case 04
Loss mask includes prompt tokens.
Loss mask includes prompt tokens · case 05
Loss mask includes prompt tokens.
Prompt tokens leak into response-only truncation · case 01
Prompt tokens leak into response-only truncation.
Prompt tokens leak into response-only truncation · case 02
Prompt tokens leak into response-only truncation.
Prompt tokens leak into response-only truncation · case 03
Prompt tokens leak into response-only truncation.
Prompt tokens leak into response-only truncation · case 04
Prompt tokens leak into response-only truncation.
Prompt tokens leak into response-only truncation · case 05
Prompt tokens leak into response-only truncation.
Flattened packed tokens retain the wrong owner indices · case 01
Flattened packed tokens retain the wrong owner indices.
Flattened packed tokens retain the wrong owner indices · case 02
Flattened packed tokens retain the wrong owner indices.
Flattened packed tokens retain the wrong owner indices · case 03
Flattened packed tokens retain the wrong owner indices.
Flattened packed tokens retain the wrong owner indices · case 04
Flattened packed tokens retain the wrong owner indices.
Flattened packed tokens retain the wrong owner indices · case 05
Flattened packed tokens retain the wrong owner indices.
Padding positions inherit the last segment owner · case 01
Padding positions inherit the last segment owner.
Padding positions inherit the last segment owner · case 02
Padding positions inherit the last segment owner.
Padding positions inherit the last segment owner · case 03
Padding positions inherit the last segment owner.
Padding positions inherit the last segment owner · case 04
Padding positions inherit the last segment owner.
Padding positions inherit the last segment owner · case 05
Padding positions inherit the last segment owner.
Future keys leak through a reversed causal comparison · case 01
Future keys leak through a reversed causal comparison.
Future keys leak through a reversed causal comparison · case 02
Future keys leak through a reversed causal comparison.
Future keys leak through a reversed causal comparison · case 03
Future keys leak through a reversed causal comparison.
Future keys leak through a reversed causal comparison · case 04
Future keys leak through a reversed causal comparison.
Future keys leak through a reversed causal comparison · case 05
Future keys leak through a reversed causal comparison.
Padded keys remain visible to active queries · case 01
Padded keys remain visible to active queries.
Padded keys remain visible to active queries · case 02
Padded keys remain visible to active queries.
Padded keys remain visible to active queries · case 03
Padded keys remain visible to active queries.
Padded keys remain visible to active queries · case 04
Padded keys remain visible to active queries.
Padded keys remain visible to active queries · case 05
Padded keys remain visible to active queries.
Padded query rows can attend to real keys · case 01
Padded query rows can attend to real keys.
Padded query rows can attend to real keys · case 02
Padded query rows can attend to real keys.
Padded query rows can attend to real keys · case 03
Padded query rows can attend to real keys.
Padded query rows can attend to real keys · case 04
Padded query rows can attend to real keys.
Padded query rows can attend to real keys · case 05
Padded query rows can attend to real keys.
Sliding window omits the oldest key within its span · case 01
Sliding window omits the oldest key within its span.
Sliding window omits the oldest key within its span · case 02
Sliding window omits the oldest key within its span.
Sliding window omits the oldest key within its span · case 03
Sliding window omits the oldest key within its span.
Sliding window omits the oldest key within its span · case 04
Sliding window omits the oldest key within its span.
Sliding window omits the oldest key within its span · case 05
Sliding window omits the oldest key within its span.
Attention crosses independent block boundaries · case 01
Attention crosses independent block boundaries.
Attention crosses independent block boundaries · case 02
Attention crosses independent block boundaries.
Attention crosses independent block boundaries · case 03
Attention crosses independent block boundaries.
Attention crosses independent block boundaries · case 04
Attention crosses independent block boundaries.
Attention crosses independent block boundaries · case 05
Attention crosses independent block boundaries.
Bidirectional block attention is accidentally causal · case 01
Bidirectional block attention is accidentally causal.
Bidirectional block attention is accidentally causal · case 02
Bidirectional block attention is accidentally causal.
Bidirectional block attention is accidentally causal · case 03
Bidirectional block attention is accidentally causal.
Bidirectional block attention is accidentally causal · case 04
Bidirectional block attention is accidentally causal.
Bidirectional block attention is accidentally causal · case 05
Bidirectional block attention is accidentally causal.
Prefix tokens cannot see the rest of their prefix · case 01
Prefix tokens cannot see the rest of their prefix.
Prefix tokens cannot see the rest of their prefix · case 02
Prefix tokens cannot see the rest of their prefix.
Prefix tokens cannot see the rest of their prefix · case 03
Prefix tokens cannot see the rest of their prefix.
Prefix tokens cannot see the rest of their prefix · case 04
Prefix tokens cannot see the rest of their prefix.
Prefix tokens cannot see the rest of their prefix · case 05
Prefix tokens cannot see the rest of their prefix.
Cached rectangular attention uses local instead of absolute positions · case 01
Cached rectangular attention uses local instead of absolute positions.
Cached rectangular attention uses local instead of absolute positions · case 02
Cached rectangular attention uses local instead of absolute positions.
Cached rectangular attention uses local instead of absolute positions · case 03
Cached rectangular attention uses local instead of absolute positions.
Cached rectangular attention uses local instead of absolute positions · case 04
Cached rectangular attention uses local instead of absolute positions.
Cached rectangular attention uses local instead of absolute positions · case 05
Cached rectangular attention uses local instead of absolute positions.
Cross-attention mask swaps source and query dimensions · case 01
Cross-attention mask swaps source and query dimensions.
Cross-attention mask swaps source and query dimensions · case 02
Cross-attention mask swaps source and query dimensions.
Cross-attention mask swaps source and query dimensions · case 03
Cross-attention mask swaps source and query dimensions.
Cross-attention mask swaps source and query dimensions · case 04
Cross-attention mask swaps source and query dimensions.
Cross-attention mask swaps source and query dimensions · case 05
Cross-attention mask swaps source and query dimensions.
Causal ordering hides valid encoder states in cross-attention · case 01
Causal ordering hides valid encoder states in cross-attention.
Causal ordering hides valid encoder states in cross-attention · case 02
Causal ordering hides valid encoder states in cross-attention.
Causal ordering hides valid encoder states in cross-attention · case 03
Causal ordering hides valid encoder states in cross-attention.
Causal ordering hides valid encoder states in cross-attention · case 04
Causal ordering hides valid encoder states in cross-attention.
Causal ordering hides valid encoder states in cross-attention · case 05
Causal ordering hides valid encoder states in cross-attention.
Dilated attention includes every intermediate key · case 01
Dilated attention includes every intermediate key.
Dilated attention includes every intermediate key · case 02
Dilated attention includes every intermediate key.
Dilated attention includes every intermediate key · case 03
Dilated attention includes every intermediate key.
Dilated attention includes every intermediate key · case 04
Dilated attention includes every intermediate key.
Dilated attention includes every intermediate key · case 05
Dilated attention includes every intermediate key.
Global tokens are incorrectly limited by local window · case 01
Global tokens are incorrectly limited by local window.
Global tokens are incorrectly limited by local window · case 02
Global tokens are incorrectly limited by local window.
Global tokens are incorrectly limited by local window · case 03
Global tokens are incorrectly limited by local window.
Global tokens are incorrectly limited by local window · case 04
Global tokens are incorrectly limited by local window.
Global tokens are incorrectly limited by local window · case 05
Global tokens are incorrectly limited by local window.
INSPECTABLE BY DESIGN
Every result has a runnable source.
Runnable implementations with recorded outputs, source hashes, and explicit contracts. Related variants share a failure mechanism and belong together in evaluation splits.
Read the methodology ↗