Understand the failure.
Verify the repair.
Small, reproducible software failures. The broken implementation, the fix that didn’t work, and the one that passed—preserved together.
Explore the cases ↓How results are verified ↗WHAT THE ARCHIVE CONTAINS
100840 executable cases. 20168 are open.
Every case records the implementation that fails, the fix that did not work, and the repair that passed its checks—with recorded outputs and source hashes. This release adds 100840 cases across 20168 failure mechanisms and 254 domains.
The open tier gives you the failure and the unsuccessful fix for one case in every mechanism. The remaining 80672 cases, 5 variants per mechanism, are member-only: the verified repair, its recorded checks, and the full fixture suite are held in the member archive. Read the methodology ↗
A RECORD OF WHAT WENT WRONG
Browse the archive / 100840
Nested required fields are checked on the outer object · case 01
Nested required fields are checked on the outer object.
Nested required fields are checked on the outer object · case 02
Nested required fields are checked on the outer object.
Nested required fields are checked on the outer object · case 03
Nested required fields are checked on the outer object.
Nested required fields are checked on the outer object · case 04
Nested required fields are checked on the outer object.
Nested required fields are checked on the outer object · case 05
Nested required fields are checked on the outer object.
Default insertion overwrites explicit null · case 01
Default insertion overwrites explicit null.
Default insertion overwrites explicit null · case 02
Default insertion overwrites explicit null.
Default insertion overwrites explicit null · case 03
Default insertion overwrites explicit null.
Default insertion overwrites explicit null · case 04
Default insertion overwrites explicit null.
Default insertion overwrites explicit null · case 05
Default insertion overwrites explicit null.
Nullable field rejects null before applying its base type · case 01
Nullable field rejects null before applying its base type.
Nullable field rejects null before applying its base type · case 02
Nullable field rejects null before applying its base type.
Nullable field rejects null before applying its base type · case 03
Nullable field rejects null before applying its base type.
Nullable field rejects null before applying its base type · case 04
Nullable field rejects null before applying its base type.
Nullable field rejects null before applying its base type · case 05
Nullable field rejects null before applying its base type.
One-of union accepts values matching both branches · case 01
One-of union accepts values matching both branches.
One-of union accepts values matching both branches · case 02
One-of union accepts values matching both branches.
One-of union accepts values matching both branches · case 03
One-of union accepts values matching both branches.
One-of union accepts values matching both branches · case 04
One-of union accepts values matching both branches.
One-of union accepts values matching both branches · case 05
One-of union accepts values matching both branches.
Any-of union requires every branch to match · case 01
Any-of union requires every branch to match.
Any-of union requires every branch to match · case 02
Any-of union requires every branch to match.
Any-of union requires every branch to match · case 03
Any-of union requires every branch to match.
Any-of union requires every branch to match · case 04
Any-of union requires every branch to match.
Any-of union requires every branch to match · case 05
Any-of union requires every branch to match.
All-of schema accepts when one constraint fails · case 01
All-of schema accepts when one constraint fails.
All-of schema accepts when one constraint fails · case 02
All-of schema accepts when one constraint fails.
All-of schema accepts when one constraint fails · case 03
All-of schema accepts when one constraint fails.
All-of schema accepts when one constraint fails · case 04
All-of schema accepts when one constraint fails.
All-of schema accepts when one constraint fails · case 05
All-of schema accepts when one constraint fails.
Dependent required condition is triggered by absent key · case 01
Dependent required condition is triggered by absent key.
Dependent required condition is triggered by absent key · case 02
Dependent required condition is triggered by absent key.
Dependent required condition is triggered by absent key · case 03
Dependent required condition is triggered by absent key.
Dependent required condition is triggered by absent key · case 04
Dependent required condition is triggered by absent key.
Dependent required condition is triggered by absent key · case 05
Dependent required condition is triggered by absent key.
Property name rule checks values instead of keys · case 01
Property name rule checks values instead of keys.
Property name rule checks values instead of keys · case 02
Property name rule checks values instead of keys.
Property name rule checks values instead of keys · case 03
Property name rule checks values instead of keys.
Property name rule checks values instead of keys · case 04
Property name rule checks values instead of keys.
Property name rule checks values instead of keys · case 05
Property name rule checks values instead of keys.
Workflow: terminal state is ignored while budget remains · case 01
A completed or failed run is scheduled for another step.
Workflow: terminal state is ignored while budget remains · case 02
A completed or failed run is scheduled for another step.
Workflow: terminal state is ignored while budget remains · case 03
A completed or failed run is scheduled for another step.
Workflow: terminal state is ignored while budget remains · case 04
A completed or failed run is scheduled for another step.
Workflow: terminal state is ignored while budget remains · case 05
A completed or failed run is scheduled for another step.
Workflow: final answer is accepted with tools still pending · case 01
A workflow closes while an issued tool request has no result.
Workflow: final answer is accepted with tools still pending · case 02
A workflow closes while an issued tool request has no result.
Workflow: final answer is accepted with tools still pending · case 03
A workflow closes while an issued tool request has no result.
Workflow: final answer is accepted with tools still pending · case 04
A workflow closes while an issued tool request has no result.
Workflow: final answer is accepted with tools still pending · case 05
A workflow closes while an issued tool request has no result.
Workflow: unrelated tool result clears the outstanding request · case 01
A run resumes after receiving a result for a different call identifier.
Workflow: unrelated tool result clears the outstanding request · case 02
A run resumes after receiving a result for a different call identifier.
Workflow: unrelated tool result clears the outstanding request · case 03
A run resumes after receiving a result for a different call identifier.
Workflow: unrelated tool result clears the outstanding request · case 04
A run resumes after receiving a result for a different call identifier.
Workflow: unrelated tool result clears the outstanding request · case 05
A run resumes after receiving a result for a different call identifier.
Workflow: progress is measured over the wrong window · case 01
A productive run is stopped even though a progress event is recent.
Workflow: progress is measured over the wrong window · case 02
A productive run is stopped even though a progress event is recent.
Workflow: progress is measured over the wrong window · case 03
A productive run is stopped even though a progress event is recent.
Workflow: progress is measured over the wrong window · case 04
A productive run is stopped even though a progress event is recent.
Workflow: progress is measured over the wrong window · case 05
A productive run is stopped even though a progress event is recent.
Workflow: tool-call allowance counts issued calls only · case 01
The loop issues too many tools when some requests are still in flight.
Workflow: tool-call allowance counts issued calls only · case 02
The loop issues too many tools when some requests are still in flight.
Workflow: tool-call allowance counts issued calls only · case 03
The loop issues too many tools when some requests are still in flight.
Workflow: tool-call allowance counts issued calls only · case 04
The loop issues too many tools when some requests are still in flight.
Workflow: tool-call allowance counts issued calls only · case 05
The loop issues too many tools when some requests are still in flight.
Workflow: a productive turn fails to reset the stall counter · case 01
The workflow stops after separated idle turns even though progress reset the streak.
Workflow: a productive turn fails to reset the stall counter · case 02
The workflow stops after separated idle turns even though progress reset the streak.
Workflow: a productive turn fails to reset the stall counter · case 03
The workflow stops after separated idle turns even though progress reset the streak.
Workflow: a productive turn fails to reset the stall counter · case 04
The workflow stops after separated idle turns even though progress reset the streak.
Workflow: a productive turn fails to reset the stall counter · case 05
The workflow stops after separated idle turns even though progress reset the streak.
Workflow: empty final payload is treated as completion · case 01
A blank assistant payload terminates the workflow as a successful answer.
Workflow: empty final payload is treated as completion · case 02
A blank assistant payload terminates the workflow as a successful answer.
Workflow: empty final payload is treated as completion · case 03
A blank assistant payload terminates the workflow as a successful answer.
Workflow: empty final payload is treated as completion · case 04
A blank assistant payload terminates the workflow as a successful answer.
Workflow: empty final payload is treated as completion · case 05
A blank assistant payload terminates the workflow as a successful answer.
Workflow: workflow completes when its first branch finishes · case 01
The run closes while another required branch is still active.
Workflow: workflow completes when its first branch finishes · case 02
The run closes while another required branch is still active.
Workflow: workflow completes when its first branch finishes · case 03
The run closes while another required branch is still active.
Workflow: workflow completes when its first branch finishes · case 04
The run closes while another required branch is still active.
Workflow: workflow completes when its first branch finishes · case 05
The run closes while another required branch is still active.
Workflow: completion is delayed after the last result drains the queue · case 01
The final result drains the queue, but the completion check still sees the pre-result count.
Workflow: completion is delayed after the last result drains the queue · case 02
The final result drains the queue, but the completion check still sees the pre-result count.
Workflow: completion is delayed after the last result drains the queue · case 03
The final result drains the queue, but the completion check still sees the pre-result count.
Workflow: completion is delayed after the last result drains the queue · case 04
The final result drains the queue, but the completion check still sees the pre-result count.
Workflow: completion is delayed after the last result drains the queue · case 05
The final result drains the queue, but the completion check still sees the pre-result count.
Workflow: a turn starts without reserving its maximum output budget · case 01
A turn begins even though its required reserve would exceed the run budget.
Workflow: a turn starts without reserving its maximum output budget · case 02
A turn begins even though its required reserve would exceed the run budget.
Workflow: a turn starts without reserving its maximum output budget · case 03
A turn begins even though its required reserve would exceed the run budget.
Workflow: a turn starts without reserving its maximum output budget · case 04
A turn begins even though its required reserve would exceed the run budget.
Workflow: a turn starts without reserving its maximum output budget · case 05
A turn begins even though its required reserve would exceed the run budget.
Workflow: a changed state is mistaken for a repeated loop state · case 01
A healthy workflow is halted because one state field changed.
Workflow: a changed state is mistaken for a repeated loop state · case 02
A healthy workflow is halted because one state field changed.
Workflow: a changed state is mistaken for a repeated loop state · case 03
A healthy workflow is halted because one state field changed.
Workflow: a changed state is mistaken for a repeated loop state · case 04
A healthy workflow is halted because one state field changed.
Workflow: a changed state is mistaken for a repeated loop state · case 05
A healthy workflow is halted because one state field changed.
Workflow: stale progress count survives a productive action · case 01
An old no-progress count triggers termination after a successful action.
Workflow: stale progress count survives a productive action · case 02
An old no-progress count triggers termination after a successful action.
Workflow: stale progress count survives a productive action · case 03
An old no-progress count triggers termination after a successful action.
Workflow: stale progress count survives a productive action · case 04
An old no-progress count triggers termination after a successful action.
Workflow: stale progress count survives a productive action · case 05
An old no-progress count triggers termination after a successful action.
INSPECTABLE BY DESIGN
Every result has a runnable source.
Runnable implementations with recorded outputs, source hashes, and explicit contracts. Related variants share a failure mechanism and belong together in evaluation splits.
Read the methodology ↗