Understand the failure.
Verify the repair.
Small, reproducible software failures. The broken implementation, the fix that didn’t work, and the one that passed—preserved together.
Explore the cases ↓How results are verified ↗WHAT THE ARCHIVE CONTAINS
100840 executable cases. 20168 are open.
Every case records the implementation that fails, the fix that did not work, and the repair that passed its checks—with recorded outputs and source hashes. This release adds 100840 cases across 20168 failure mechanisms and 254 domains.
The open tier gives you the failure and the unsuccessful fix for one case in every mechanism. The remaining 80672 cases, 5 variants per mechanism, are member-only: the verified repair, its recorded checks, and the full fixture suite are held in the member archive. Read the methodology ↗
A RECORD OF WHAT WENT WRONG
Browse the archive / 100840
Decoding: Committed prefix is reversed before output · case 01
Committed prefix is reversed before output.
Decoding: Committed prefix is reversed before output · case 02
Committed prefix is reversed before output.
Decoding: Committed prefix is reversed before output · case 03
Committed prefix is reversed before output.
Decoding: Committed prefix is reversed before output · case 04
Committed prefix is reversed before output.
Decoding: Committed prefix is reversed before output · case 05
Committed prefix is reversed before output.
Decoding: Only the final request contributes to the batch acceptance total · case 01
Only the final request contributes to the batch acceptance total.
Decoding: Only the final request contributes to the batch acceptance total · case 02
Only the final request contributes to the batch acceptance total.
Decoding: Only the final request contributes to the batch acceptance total · case 03
Only the final request contributes to the batch acceptance total.
Decoding: Only the final request contributes to the batch acceptance total · case 04
Only the final request contributes to the batch acceptance total.
Decoding: Only the final request contributes to the batch acceptance total · case 05
Only the final request contributes to the batch acceptance total.
Decoding: Unaccepted proposal count is inflated by a replacement slot · case 01
Unaccepted proposal count is inflated by a replacement slot.
Decoding: Unaccepted proposal count is inflated by a replacement slot · case 02
Unaccepted proposal count is inflated by a replacement slot.
Decoding: Unaccepted proposal count is inflated by a replacement slot · case 03
Unaccepted proposal count is inflated by a replacement slot.
Decoding: Unaccepted proposal count is inflated by a replacement slot · case 04
Unaccepted proposal count is inflated by a replacement slot.
Decoding: Unaccepted proposal count is inflated by a replacement slot · case 05
Unaccepted proposal count is inflated by a replacement slot.
Decoding: Flags after the first rejection are counted as accepted · case 01
Flags after the first rejection are counted as accepted.
Decoding: Flags after the first rejection are counted as accepted · case 02
Flags after the first rejection are counted as accepted.
Decoding: Flags after the first rejection are counted as accepted · case 03
Flags after the first rejection are counted as accepted.
Decoding: Flags after the first rejection are counted as accepted · case 04
Flags after the first rejection are counted as accepted.
Decoding: Flags after the first rejection are counted as accepted · case 05
Flags after the first rejection are counted as accepted.
Decoding: Replacement is emitted before the committed prefix · case 01
Replacement is emitted before the committed prefix.
Decoding: Replacement is emitted before the committed prefix · case 02
Replacement is emitted before the committed prefix.
Decoding: Replacement is emitted before the committed prefix · case 03
Replacement is emitted before the committed prefix.
Decoding: Replacement is emitted before the committed prefix · case 04
Replacement is emitted before the committed prefix.
Decoding: Replacement is emitted before the committed prefix · case 05
Replacement is emitted before the committed prefix.
Decoding: Mismatch report returns the draft token instead of the verified token · case 01
Mismatch report returns the draft token instead of the verified token.
Decoding: Mismatch report returns the draft token instead of the verified token · case 02
Mismatch report returns the draft token instead of the verified token.
Decoding: Mismatch report returns the draft token instead of the verified token · case 03
Mismatch report returns the draft token instead of the verified token.
Decoding: Mismatch report returns the draft token instead of the verified token · case 04
Mismatch report returns the draft token instead of the verified token.
Decoding: Mismatch report returns the draft token instead of the verified token · case 05
Mismatch report returns the draft token instead of the verified token.
Decoding: Committed score includes the rejected token score · case 01
Committed score includes the rejected token score.
Decoding: Committed score includes the rejected token score · case 02
Committed score includes the rejected token score.
Decoding: Committed score includes the rejected token score · case 03
Committed score includes the rejected token score.
Decoding: Committed score includes the rejected token score · case 04
Committed score includes the rejected token score.
Decoding: Committed score includes the rejected token score · case 05
Committed score includes the rejected token score.
Decoding: Per-step acceptance record marks the rejected position as accepted · case 01
Per-step acceptance record marks the rejected position as accepted.
Decoding: Per-step acceptance record marks the rejected position as accepted · case 02
Per-step acceptance record marks the rejected position as accepted.
Decoding: Per-step acceptance record marks the rejected position as accepted · case 03
Per-step acceptance record marks the rejected position as accepted.
Decoding: Per-step acceptance record marks the rejected position as accepted · case 04
Per-step acceptance record marks the rejected position as accepted.
Decoding: Per-step acceptance record marks the rejected position as accepted · case 05
Per-step acceptance record marks the rejected position as accepted.
Output stopping: Configured stop order overrides earliest stream position · case 01
Configured stop order overrides earliest stream position.
Output stopping: Configured stop order overrides earliest stream position · case 02
Configured stop order overrides earliest stream position.
Output stopping: Configured stop order overrides earliest stream position · case 03
Configured stop order overrides earliest stream position.
Output stopping: Configured stop order overrides earliest stream position · case 04
Configured stop order overrides earliest stream position.
Output stopping: Configured stop order overrides earliest stream position · case 05
Configured stop order overrides earliest stream position.
Output stopping: Tied stop matches choose the longest sequence · case 01
Tied stop matches choose the longest sequence.
Output stopping: Tied stop matches choose the longest sequence · case 02
Tied stop matches choose the longest sequence.
Output stopping: Tied stop matches choose the longest sequence · case 03
Tied stop matches choose the longest sequence.
Output stopping: Tied stop matches choose the longest sequence · case 04
Tied stop matches choose the longest sequence.
Output stopping: Tied stop matches choose the longest sequence · case 05
Tied stop matches choose the longest sequence.
Output stopping: Token ID zero is treated as a disabled stop token · case 01
Token ID zero is treated as a disabled stop token.
Output stopping: Token ID zero is treated as a disabled stop token · case 02
Token ID zero is treated as a disabled stop token.
Output stopping: Token ID zero is treated as a disabled stop token · case 03
Token ID zero is treated as a disabled stop token.
Output stopping: Token ID zero is treated as a disabled stop token · case 04
Token ID zero is treated as a disabled stop token.
Output stopping: Token ID zero is treated as a disabled stop token · case 05
Token ID zero is treated as a disabled stop token.
Output stopping: Prompt tokens consume the new-token budget · case 01
Prompt tokens consume the new-token budget.
Output stopping: Prompt tokens consume the new-token budget · case 02
Prompt tokens consume the new-token budget.
Output stopping: Prompt tokens consume the new-token budget · case 03
Prompt tokens consume the new-token budget.
Output stopping: Prompt tokens consume the new-token budget · case 04
Prompt tokens consume the new-token budget.
Output stopping: Prompt tokens consume the new-token budget · case 05
Prompt tokens consume the new-token budget.
Output stopping: Prompt length is omitted from a total context cap · case 01
Prompt length is omitted from a total context cap.
Output stopping: Prompt length is omitted from a total context cap · case 02
Prompt length is omitted from a total context cap.
Output stopping: Prompt length is omitted from a total context cap · case 03
Prompt length is omitted from a total context cap.
Output stopping: Prompt length is omitted from a total context cap · case 04
Prompt length is omitted from a total context cap.
Output stopping: Prompt length is omitted from a total context cap · case 05
Prompt length is omitted from a total context cap.
Output stopping: Stop matches are honored before minimum length · case 01
Stop matches are honored before minimum length.
Output stopping: Stop matches are honored before minimum length · case 02
Stop matches are honored before minimum length.
Output stopping: Stop matches are honored before minimum length · case 03
Stop matches are honored before minimum length.
Output stopping: Stop matches are honored before minimum length · case 04
Stop matches are honored before minimum length.
Output stopping: Stop matches are honored before minimum length · case 05
Stop matches are honored before minimum length.
Output stopping: A matched stop marker remains in returned output · case 01
A matched stop marker remains in returned output.
Output stopping: A matched stop marker remains in returned output · case 02
A matched stop marker remains in returned output.
Output stopping: A matched stop marker remains in returned output · case 03
A matched stop marker remains in returned output.
Output stopping: A matched stop marker remains in returned output · case 04
A matched stop marker remains in returned output.
Output stopping: A matched stop marker remains in returned output · case 05
A matched stop marker remains in returned output.
Output stopping: A length finish overwrites an observed stop reason · case 01
A length finish overwrites an observed stop reason.
Output stopping: A length finish overwrites an observed stop reason · case 02
A length finish overwrites an observed stop reason.
Output stopping: A length finish overwrites an observed stop reason · case 03
A length finish overwrites an observed stop reason.
Output stopping: A length finish overwrites an observed stop reason · case 04
A length finish overwrites an observed stop reason.
Output stopping: A length finish overwrites an observed stop reason · case 05
A length finish overwrites an observed stop reason.
Output stopping: EOS is classified without truncating output · case 01
EOS is classified without truncating output.
Output stopping: EOS is classified without truncating output · case 02
EOS is classified without truncating output.
Output stopping: EOS is classified without truncating output · case 03
EOS is classified without truncating output.
Output stopping: EOS is classified without truncating output · case 04
EOS is classified without truncating output.
Output stopping: EOS is classified without truncating output · case 05
EOS is classified without truncating output.
Output stopping: Streaming buffer emits partial stop prefixes too early · case 01
Streaming buffer emits partial stop prefixes too early.
Output stopping: Streaming buffer emits partial stop prefixes too early · case 02
Streaming buffer emits partial stop prefixes too early.
Output stopping: Streaming buffer emits partial stop prefixes too early · case 03
Streaming buffer emits partial stop prefixes too early.
Output stopping: Streaming buffer emits partial stop prefixes too early · case 04
Streaming buffer emits partial stop prefixes too early.
Output stopping: Streaming buffer emits partial stop prefixes too early · case 05
Streaming buffer emits partial stop prefixes too early.
Output stopping: Normal stream completion drops the held partial marker · case 01
Normal stream completion drops the held partial marker.
Output stopping: Normal stream completion drops the held partial marker · case 02
Normal stream completion drops the held partial marker.
Output stopping: Normal stream completion drops the held partial marker · case 03
Normal stream completion drops the held partial marker.
Output stopping: Normal stream completion drops the held partial marker · case 04
Normal stream completion drops the held partial marker.
Output stopping: Normal stream completion drops the held partial marker · case 05
Normal stream completion drops the held partial marker.
Output stopping: A stop sequence split across chunks is missed · case 01
A stop sequence split across chunks is missed.
Output stopping: A stop sequence split across chunks is missed · case 02
A stop sequence split across chunks is missed.
Output stopping: A stop sequence split across chunks is missed · case 03
A stop sequence split across chunks is missed.
Output stopping: A stop sequence split across chunks is missed · case 04
A stop sequence split across chunks is missed.
Output stopping: A stop sequence split across chunks is missed · case 05
A stop sequence split across chunks is missed.
INSPECTABLE BY DESIGN
Every result has a runnable source.
Runnable implementations with recorded outputs, source hashes, and explicit contracts. Related variants share a failure mechanism and belong together in evaluation splits.
Read the methodology ↗