Understand the failure.
Verify the repair.
Small, reproducible software failures. The broken implementation, the fix that didn’t work, and the one that passed—preserved together.
Explore the cases ↓How results are verified ↗WHAT THE ARCHIVE CONTAINS
100840 executable cases. 20168 are open.
Every case records the implementation that fails, the fix that did not work, and the repair that passed its checks—with recorded outputs and source hashes. This release adds 100840 cases across 20168 failure mechanisms and 254 domains.
The open tier gives you the failure and the unsuccessful fix for one case in every mechanism. The remaining 80672 cases, 5 variants per mechanism, are member-only: the verified repair, its recorded checks, and the full fixture suite are held in the member archive. Read the methodology ↗
A RECORD OF WHAT WENT WRONG
Browse the archive / 100840
Token sampling: Token bias is applied after repetition adjustment · case 01
Token bias is applied after repetition adjustment.
Token sampling: Token bias is applied after repetition adjustment · case 02
Token bias is applied after repetition adjustment.
Token sampling: Token bias is applied after repetition adjustment · case 03
Token bias is applied after repetition adjustment.
Token sampling: Token bias is applied after repetition adjustment · case 04
Token bias is applied after repetition adjustment.
Token sampling: Token bias is applied after repetition adjustment · case 05
Token bias is applied after repetition adjustment.
Token sampling: Bad-word sequence filter checks only the final token · case 01
Bad-word sequence filter checks only the final token.
Token sampling: Bad-word sequence filter checks only the final token · case 02
Bad-word sequence filter checks only the final token.
Token sampling: Bad-word sequence filter checks only the final token · case 03
Bad-word sequence filter checks only the final token.
Token sampling: Bad-word sequence filter checks only the final token · case 04
Bad-word sequence filter checks only the final token.
Token sampling: Bad-word sequence filter checks only the final token · case 05
Bad-word sequence filter checks only the final token.
Token sampling: No-repeat n-gram filter bans every token seen in an n-gram · case 01
No-repeat n-gram filter bans every token seen in an n-gram.
Token sampling: No-repeat n-gram filter bans every token seen in an n-gram · case 02
No-repeat n-gram filter bans every token seen in an n-gram.
Token sampling: No-repeat n-gram filter bans every token seen in an n-gram · case 03
No-repeat n-gram filter bans every token seen in an n-gram.
Token sampling: No-repeat n-gram filter bans every token seen in an n-gram · case 04
No-repeat n-gram filter bans every token seen in an n-gram.
Token sampling: No-repeat n-gram filter bans every token seen in an n-gram · case 05
No-repeat n-gram filter bans every token seen in an n-gram.
Token sampling: Adaptive top-a uses an absolute rather than relative gap · case 01
Adaptive top-a uses an absolute rather than relative gap.
Token sampling: Adaptive top-a uses an absolute rather than relative gap · case 02
Adaptive top-a uses an absolute rather than relative gap.
Token sampling: Adaptive top-a uses an absolute rather than relative gap · case 03
Adaptive top-a uses an absolute rather than relative gap.
Token sampling: Adaptive top-a uses an absolute rather than relative gap · case 04
Adaptive top-a uses an absolute rather than relative gap.
Token sampling: Adaptive top-a uses an absolute rather than relative gap · case 05
Adaptive top-a uses an absolute rather than relative gap.
Token sampling: Eta cutoff compares log probability to a probability threshold · case 01
Eta cutoff compares log probability to a probability threshold.
Token sampling: Eta cutoff compares log probability to a probability threshold · case 02
Eta cutoff compares log probability to a probability threshold.
Token sampling: Eta cutoff compares log probability to a probability threshold · case 03
Eta cutoff compares log probability to a probability threshold.
Token sampling: Eta cutoff compares log probability to a probability threshold · case 04
Eta cutoff compares log probability to a probability threshold.
Token sampling: Eta cutoff compares log probability to a probability threshold · case 05
Eta cutoff compares log probability to a probability threshold.
Token sampling: Top-k probability mass is measured before candidate restriction · case 01
Top-k probability mass is measured before candidate restriction.
Token sampling: Top-k probability mass is measured before candidate restriction · case 02
Top-k probability mass is measured before candidate restriction.
Token sampling: Top-k probability mass is measured before candidate restriction · case 03
Top-k probability mass is measured before candidate restriction.
Token sampling: Top-k probability mass is measured before candidate restriction · case 04
Top-k probability mass is measured before candidate restriction.
Token sampling: Top-k probability mass is measured before candidate restriction · case 05
Top-k probability mass is measured before candidate restriction.
Token sampling: Tail-free pruning measures drops from the unsorted vocabulary · case 01
Tail-free pruning measures drops from the unsorted vocabulary.
Token sampling: Tail-free pruning measures drops from the unsorted vocabulary · case 02
Tail-free pruning measures drops from the unsorted vocabulary.
Token sampling: Tail-free pruning measures drops from the unsorted vocabulary · case 03
Tail-free pruning measures drops from the unsorted vocabulary.
Token sampling: Tail-free pruning measures drops from the unsorted vocabulary · case 04
Tail-free pruning measures drops from the unsorted vocabulary.
Token sampling: Tail-free pruning measures drops from the unsorted vocabulary · case 05
Tail-free pruning measures drops from the unsorted vocabulary.
Token sampling: Prompt history is included in repetition adjustment · case 01
Prompt history is included in repetition adjustment.
Token sampling: Prompt history is included in repetition adjustment · case 02
Prompt history is included in repetition adjustment.
Token sampling: Prompt history is included in repetition adjustment · case 03
Prompt history is included in repetition adjustment.
Token sampling: Prompt history is included in repetition adjustment · case 04
Prompt history is included in repetition adjustment.
Token sampling: Prompt history is included in repetition adjustment · case 05
Prompt history is included in repetition adjustment.
Beam search: child score drops its parent history · case 01
child score drops its parent history.
Beam search: child score drops its parent history · case 02
child score drops its parent history.
Beam search: child score drops its parent history · case 03
child score drops its parent history.
Beam search: child score drops its parent history · case 04
child score drops its parent history.
Beam search: child score drops its parent history · case 05
child score drops its parent history.
Beam search: backtrace follows pre-prune beam slots · case 01
backtrace follows pre-prune beam slots.
Beam search: backtrace follows pre-prune beam slots · case 02
backtrace follows pre-prune beam slots.
Beam search: backtrace follows pre-prune beam slots · case 03
backtrace follows pre-prune beam slots.
Beam search: backtrace follows pre-prune beam slots · case 04
backtrace follows pre-prune beam slots.
Beam search: backtrace follows pre-prune beam slots · case 05
backtrace follows pre-prune beam slots.
Beam search: beam width is enforced independently per batch item · case 01
beam width is enforced independently per batch item.
Beam search: beam width is enforced independently per batch item · case 02
beam width is enforced independently per batch item.
Beam search: beam width is enforced independently per batch item · case 03
beam width is enforced independently per batch item.
Beam search: beam width is enforced independently per batch item · case 04
beam width is enforced independently per batch item.
Beam search: beam width is enforced independently per batch item · case 05
beam width is enforced independently per batch item.
Beam search: candidate budget is shared before per-parent expansion · case 01
candidate budget is shared before per-parent expansion.
Beam search: candidate budget is shared before per-parent expansion · case 02
candidate budget is shared before per-parent expansion.
Beam search: candidate budget is shared before per-parent expansion · case 03
candidate budget is shared before per-parent expansion.
Beam search: candidate budget is shared before per-parent expansion · case 04
candidate budget is shared before per-parent expansion.
Beam search: candidate budget is shared before per-parent expansion · case 05
candidate budget is shared before per-parent expansion.
Beam search: end marker is expanded as an ordinary live token · case 01
end marker is expanded as an ordinary live token.
Beam search: end marker is expanded as an ordinary live token · case 02
end marker is expanded as an ordinary live token.
Beam search: end marker is expanded as an ordinary live token · case 03
end marker is expanded as an ordinary live token.
Beam search: end marker is expanded as an ordinary live token · case 04
end marker is expanded as an ordinary live token.
Beam search: end marker is expanded as an ordinary live token · case 05
end marker is expanded as an ordinary live token.
Beam search: terminal score omits the end-marker contribution · case 01
terminal score omits the end-marker contribution.
Beam search: terminal score omits the end-marker contribution · case 02
terminal score omits the end-marker contribution.
Beam search: terminal score omits the end-marker contribution · case 03
terminal score omits the end-marker contribution.
Beam search: terminal score omits the end-marker contribution · case 04
terminal score omits the end-marker contribution.
Beam search: terminal score omits the end-marker contribution · case 05
terminal score omits the end-marker contribution.
Beam search: finished hypotheses are discarded at the next timestep · case 01
finished hypotheses are discarded at the next timestep.
Beam search: finished hypotheses are discarded at the next timestep · case 02
finished hypotheses are discarded at the next timestep.
Beam search: finished hypotheses are discarded at the next timestep · case 03
finished hypotheses are discarded at the next timestep.
Beam search: finished hypotheses are discarded at the next timestep · case 04
finished hypotheses are discarded at the next timestep.
Beam search: finished hypotheses are discarded at the next timestep · case 05
finished hypotheses are discarded at the next timestep.
Beam search: beam pruning ranks raw totals instead of normalized totals · case 01
beam pruning ranks raw totals instead of normalized totals.
Beam search: beam pruning ranks raw totals instead of normalized totals · case 02
beam pruning ranks raw totals instead of normalized totals.
Beam search: beam pruning ranks raw totals instead of normalized totals · case 03
beam pruning ranks raw totals instead of normalized totals.
Beam search: beam pruning ranks raw totals instead of normalized totals · case 04
beam pruning ranks raw totals instead of normalized totals.
Beam search: beam pruning ranks raw totals instead of normalized totals · case 05
beam pruning ranks raw totals instead of normalized totals.
Beam search: length penalty counts prompt tokens as generated output · case 01
length penalty counts prompt tokens as generated output.
Beam search: length penalty counts prompt tokens as generated output · case 02
length penalty counts prompt tokens as generated output.
Beam search: length penalty counts prompt tokens as generated output · case 03
length penalty counts prompt tokens as generated output.
Beam search: length penalty counts prompt tokens as generated output · case 04
length penalty counts prompt tokens as generated output.
Beam search: length penalty counts prompt tokens as generated output · case 05
length penalty counts prompt tokens as generated output.
Beam search: length penalty favors longer paths under a penalty contract · case 01
length penalty favors longer paths under a penalty contract.
Beam search: length penalty favors longer paths under a penalty contract · case 02
length penalty favors longer paths under a penalty contract.
Beam search: length penalty favors longer paths under a penalty contract · case 03
length penalty favors longer paths under a penalty contract.
Beam search: length penalty favors longer paths under a penalty contract · case 04
length penalty favors longer paths under a penalty contract.
Beam search: length penalty favors longer paths under a penalty contract · case 05
length penalty favors longer paths under a penalty contract.
Beam search: length normalization changes accumulated search scores · case 01
length normalization changes accumulated search scores.
Beam search: length normalization changes accumulated search scores · case 02
length normalization changes accumulated search scores.
Beam search: length normalization changes accumulated search scores · case 03
length normalization changes accumulated search scores.
Beam search: length normalization changes accumulated search scores · case 04
length normalization changes accumulated search scores.
Beam search: length normalization changes accumulated search scores · case 05
length normalization changes accumulated search scores.
Beam search: search stops before a live path can beat the finished set · case 01
search stops before a live path can beat the finished set.
Beam search: search stops before a live path can beat the finished set · case 02
search stops before a live path can beat the finished set.
Beam search: search stops before a live path can beat the finished set · case 03
search stops before a live path can beat the finished set.
Beam search: search stops before a live path can beat the finished set · case 04
search stops before a live path can beat the finished set.
Beam search: search stops before a live path can beat the finished set · case 05
search stops before a live path can beat the finished set.
INSPECTABLE BY DESIGN
Every result has a runnable source.
Runnable implementations with recorded outputs, source hashes, and explicit contracts. Related variants share a failure mechanism and belong together in evaluation splits.
Read the methodology ↗