Understand the failure.
Verify the repair.
Small, reproducible software failures. The broken implementation, the fix that didn’t work, and the one that passed—preserved together.
Explore the cases ↓How results are verified ↗WHAT THE ARCHIVE CONTAINS
100840 executable cases. 20168 are open.
Every case records the implementation that fails, the fix that did not work, and the repair that passed its checks—with recorded outputs and source hashes. This release adds 100840 cases across 20168 failure mechanisms and 254 domains.
The open tier gives you the failure and the unsuccessful fix for one case in every mechanism. The remaining 80672 cases, 5 variants per mechanism, are member-only: the verified repair, its recorded checks, and the full fixture suite are held in the member archive. Read the methodology ↗
A RECORD OF WHAT WENT WRONG
Browse the archive / 100840
Shifted pairs carry an off-by-one context position · case 01
Shifted pairs carry an off-by-one context position.
Shifted pairs carry an off-by-one context position · case 02
Shifted pairs carry an off-by-one context position.
Shifted pairs carry an off-by-one context position · case 03
Shifted pairs carry an off-by-one context position.
Shifted pairs carry an off-by-one context position · case 04
Shifted pairs carry an off-by-one context position.
Shifted pairs carry an off-by-one context position · case 05
Shifted pairs carry an off-by-one context position.
Target gather uses the context position instead of its successor · case 01
Target gather uses the context position instead of its successor.
Target gather uses the context position instead of its successor · case 02
Target gather uses the context position instead of its successor.
Target gather uses the context position instead of its successor · case 03
Target gather uses the context position instead of its successor.
Target gather uses the context position instead of its successor · case 04
Target gather uses the context position instead of its successor.
Target gather uses the context position instead of its successor · case 05
Target gather uses the context position instead of its successor.
Loss denominator counts inputs instead of supervised targets · case 01
Loss denominator counts inputs instead of supervised targets.
Loss denominator counts inputs instead of supervised targets · case 02
Loss denominator counts inputs instead of supervised targets.
Loss denominator counts inputs instead of supervised targets · case 03
Loss denominator counts inputs instead of supervised targets.
Loss denominator counts inputs instead of supervised targets · case 04
Loss denominator counts inputs instead of supervised targets.
Loss denominator counts inputs instead of supervised targets · case 05
Loss denominator counts inputs instead of supervised targets.
Ignored targets remain in the loss denominator · case 01
Ignored targets remain in the loss denominator.
Ignored targets remain in the loss denominator · case 02
Ignored targets remain in the loss denominator.
Ignored targets remain in the loss denominator · case 03
Ignored targets remain in the loss denominator.
Ignored targets remain in the loss denominator · case 04
Ignored targets remain in the loss denominator.
Ignored targets remain in the loss denominator · case 05
Ignored targets remain in the loss denominator.
A boundary input predicts the first token of the next segment · case 01
A boundary input predicts the first token of the next segment.
A boundary input predicts the first token of the next segment · case 02
A boundary input predicts the first token of the next segment.
A boundary input predicts the first token of the next segment · case 03
A boundary input predicts the first token of the next segment.
A boundary input predicts the first token of the next segment · case 04
A boundary input predicts the first token of the next segment.
A boundary input predicts the first token of the next segment · case 05
A boundary input predicts the first token of the next segment.
A segment start loses its first valid target pair · case 01
A segment start loses its first valid target pair.
A segment start loses its first valid target pair · case 02
A segment start loses its first valid target pair.
A segment start loses its first valid target pair · case 03
A segment start loses its first valid target pair.
A segment start loses its first valid target pair · case 04
A segment start loses its first valid target pair.
A segment start loses its first valid target pair · case 05
A segment start loses its first valid target pair.
A special separator token is predicted as ordinary content · case 01
A special separator token is predicted as ordinary content.
A special separator token is predicted as ordinary content · case 02
A special separator token is predicted as ordinary content.
A special separator token is predicted as ordinary content · case 03
A special separator token is predicted as ordinary content.
A special separator token is predicted as ordinary content · case 04
A special separator token is predicted as ordinary content.
A special separator token is predicted as ordinary content · case 05
A special separator token is predicted as ordinary content.
A separator context incorrectly suppresses the following target · case 01
A separator context incorrectly suppresses the following target.
A separator context incorrectly suppresses the following target · case 02
A separator context incorrectly suppresses the following target.
A separator context incorrectly suppresses the following target · case 03
A separator context incorrectly suppresses the following target.
A separator context incorrectly suppresses the following target · case 04
A separator context incorrectly suppresses the following target.
A separator context incorrectly suppresses the following target · case 05
A separator context incorrectly suppresses the following target.
Token sampling: Top-k drops the final retained candidate · case 01
Top-k drops the final retained candidate.
Token sampling: Top-k drops the final retained candidate · case 02
Top-k drops the final retained candidate.
Token sampling: Top-k drops the final retained candidate · case 03
Top-k drops the final retained candidate.
Token sampling: Top-k drops the final retained candidate · case 04
Top-k drops the final retained candidate.
Token sampling: Top-k drops the final retained candidate · case 05
Top-k drops the final retained candidate.
Token sampling: Nucleus filtering excludes the token that crosses p · case 01
Nucleus filtering excludes the token that crosses p.
Token sampling: Nucleus filtering excludes the token that crosses p · case 02
Nucleus filtering excludes the token that crosses p.
Token sampling: Nucleus filtering excludes the token that crosses p · case 03
Nucleus filtering excludes the token that crosses p.
Token sampling: Nucleus filtering excludes the token that crosses p · case 04
Nucleus filtering excludes the token that crosses p.
Token sampling: Nucleus filtering excludes the token that crosses p · case 05
Nucleus filtering excludes the token that crosses p.
Token sampling: Nucleus threshold is applied to unnormalized weights · case 01
Nucleus threshold is applied to unnormalized weights.
Token sampling: Nucleus threshold is applied to unnormalized weights · case 02
Nucleus threshold is applied to unnormalized weights.
Token sampling: Nucleus threshold is applied to unnormalized weights · case 03
Nucleus threshold is applied to unnormalized weights.
Token sampling: Nucleus threshold is applied to unnormalized weights · case 04
Nucleus threshold is applied to unnormalized weights.
Token sampling: Nucleus threshold is applied to unnormalized weights · case 05
Nucleus threshold is applied to unnormalized weights.
Token sampling: Minimum-p cutoff uses an absolute probability floor · case 01
Minimum-p cutoff uses an absolute probability floor.
Token sampling: Minimum-p cutoff uses an absolute probability floor · case 02
Minimum-p cutoff uses an absolute probability floor.
Token sampling: Minimum-p cutoff uses an absolute probability floor · case 03
Minimum-p cutoff uses an absolute probability floor.
Token sampling: Minimum-p cutoff uses an absolute probability floor · case 04
Minimum-p cutoff uses an absolute probability floor.
Token sampling: Minimum-p cutoff uses an absolute probability floor · case 05
Minimum-p cutoff uses an absolute probability floor.
Token sampling: Epsilon cutoff ignores the candidate count · case 01
Epsilon cutoff ignores the candidate count.
Token sampling: Epsilon cutoff ignores the candidate count · case 02
Epsilon cutoff ignores the candidate count.
Token sampling: Epsilon cutoff ignores the candidate count · case 03
Epsilon cutoff ignores the candidate count.
Token sampling: Epsilon cutoff ignores the candidate count · case 04
Epsilon cutoff ignores the candidate count.
Token sampling: Epsilon cutoff ignores the candidate count · case 05
Epsilon cutoff ignores the candidate count.
Token sampling: Typical sampling ranks by raw surprisal distance · case 01
Typical sampling ranks by raw surprisal distance.
Token sampling: Typical sampling ranks by raw surprisal distance · case 02
Typical sampling ranks by raw surprisal distance.
Token sampling: Typical sampling ranks by raw surprisal distance · case 03
Typical sampling ranks by raw surprisal distance.
Token sampling: Typical sampling ranks by raw surprisal distance · case 04
Typical sampling ranks by raw surprisal distance.
Token sampling: Typical sampling ranks by raw surprisal distance · case 05
Typical sampling ranks by raw surprisal distance.
Token sampling: Repetition penalty pushes positive and negative scores alike · case 01
Repetition penalty pushes positive and negative scores alike.
Token sampling: Repetition penalty pushes positive and negative scores alike · case 02
Repetition penalty pushes positive and negative scores alike.
Token sampling: Repetition penalty pushes positive and negative scores alike · case 03
Repetition penalty pushes positive and negative scores alike.
Token sampling: Repetition penalty pushes positive and negative scores alike · case 04
Repetition penalty pushes positive and negative scores alike.
Token sampling: Repetition penalty pushes positive and negative scores alike · case 05
Repetition penalty pushes positive and negative scores alike.
Token sampling: Presence penalty scales with repeated history count · case 01
Presence penalty scales with repeated history count.
Token sampling: Presence penalty scales with repeated history count · case 02
Presence penalty scales with repeated history count.
Token sampling: Presence penalty scales with repeated history count · case 03
Presence penalty scales with repeated history count.
Token sampling: Presence penalty scales with repeated history count · case 04
Presence penalty scales with repeated history count.
Token sampling: Presence penalty scales with repeated history count · case 05
Presence penalty scales with repeated history count.
Token sampling: Frequency penalty counts only distinct history tokens · case 01
Frequency penalty counts only distinct history tokens.
Token sampling: Frequency penalty counts only distinct history tokens · case 02
Frequency penalty counts only distinct history tokens.
Token sampling: Frequency penalty counts only distinct history tokens · case 03
Frequency penalty counts only distinct history tokens.
Token sampling: Frequency penalty counts only distinct history tokens · case 04
Frequency penalty counts only distinct history tokens.
Token sampling: Frequency penalty counts only distinct history tokens · case 05
Frequency penalty counts only distinct history tokens.
Token sampling: Allowed probabilities sum below one when disallowed logits remain in the denominator · case 01
Allowed probabilities sum below one when disallowed logits remain in the denominator.
Token sampling: Allowed probabilities sum below one when disallowed logits remain in the denominator · case 02
Allowed probabilities sum below one when disallowed logits remain in the denominator.
Token sampling: Allowed probabilities sum below one when disallowed logits remain in the denominator · case 03
Allowed probabilities sum below one when disallowed logits remain in the denominator.
Token sampling: Allowed probabilities sum below one when disallowed logits remain in the denominator · case 04
Allowed probabilities sum below one when disallowed logits remain in the denominator.
Token sampling: Allowed probabilities sum below one when disallowed logits remain in the denominator · case 05
Allowed probabilities sum below one when disallowed logits remain in the denominator.
Token sampling: A banned token remains in normalized output · case 01
A banned token remains in normalized output.
Token sampling: A banned token remains in normalized output · case 02
A banned token remains in normalized output.
Token sampling: A banned token remains in normalized output · case 03
A banned token remains in normalized output.
Token sampling: A banned token remains in normalized output · case 04
A banned token remains in normalized output.
Token sampling: A banned token remains in normalized output · case 05
A banned token remains in normalized output.
Token sampling: Temperature scales normalized probabilities · case 01
Temperature scales normalized probabilities.
Token sampling: Temperature scales normalized probabilities · case 02
Temperature scales normalized probabilities.
Token sampling: Temperature scales normalized probabilities · case 03
Temperature scales normalized probabilities.
Token sampling: Temperature scales normalized probabilities · case 04
Temperature scales normalized probabilities.
Token sampling: Temperature scales normalized probabilities · case 05
Temperature scales normalized probabilities.
INSPECTABLE BY DESIGN
Every result has a runnable source.
Runnable implementations with recorded outputs, source hashes, and explicit contracts. Related variants share a failure mechanism and belong together in evaluation splits.
Read the methodology ↗