Understand the failure.
Verify the repair.
Small, reproducible software failures. The broken implementation, the fix that didn’t work, and the one that passed—preserved together.
Explore the cases ↓How results are verified ↗WHAT THE ARCHIVE CONTAINS
100840 executable cases. 20168 are open.
Every case records the implementation that fails, the fix that did not work, and the repair that passed its checks—with recorded outputs and source hashes. This release adds 100840 cases across 20168 failure mechanisms and 254 domains.
The open tier gives you the failure and the unsuccessful fix for one case in every mechanism. The remaining 80672 cases, 5 variants per mechanism, are member-only: the verified repair, its recorded checks, and the full fixture suite are held in the member archive. Read the methodology ↗
A RECORD OF WHAT WENT WRONG
Browse the archive / 100840
Workflow: cancelled workflow receives another turn · case 01
A cancelled run remains eligible for another agent turn.
Workflow: cancelled workflow receives another turn · case 02
A cancelled run remains eligible for another agent turn.
Workflow: cancelled workflow receives another turn · case 03
A cancelled run remains eligible for another agent turn.
Workflow: cancelled workflow receives another turn · case 04
A cancelled run remains eligible for another agent turn.
Workflow: cancelled workflow receives another turn · case 05
A cancelled run remains eligible for another agent turn.
Workflow: batch is settled when any tool call returns · case 01
The workflow advances after the first result of a multi-call batch.
Workflow: batch is settled when any tool call returns · case 02
The workflow advances after the first result of a multi-call batch.
Workflow: batch is settled when any tool call returns · case 03
The workflow advances after the first result of a multi-call batch.
Workflow: batch is settled when any tool call returns · case 04
The workflow advances after the first result of a multi-call batch.
Workflow: batch is settled when any tool call returns · case 05
The workflow advances after the first result of a multi-call batch.
Workflow: error limit ignores a successful turn between errors · case 01
Separated tool errors trigger a consecutive-error stop.
Workflow: error limit ignores a successful turn between errors · case 02
Separated tool errors trigger a consecutive-error stop.
Workflow: error limit ignores a successful turn between errors · case 03
Separated tool errors trigger a consecutive-error stop.
Workflow: error limit ignores a successful turn between errors · case 04
Separated tool errors trigger a consecutive-error stop.
Workflow: error limit ignores a successful turn between errors · case 05
Separated tool errors trigger a consecutive-error stop.
Workflow: budget exhaustion is reported as successful completion · case 01
A run with no remaining work is labeled completed despite having no final answer.
Workflow: budget exhaustion is reported as successful completion · case 02
A run with no remaining work is labeled completed despite having no final answer.
Workflow: budget exhaustion is reported as successful completion · case 03
A run with no remaining work is labeled completed despite having no final answer.
Workflow: budget exhaustion is reported as successful completion · case 04
A run with no remaining work is labeled completed despite having no final answer.
Workflow: budget exhaustion is reported as successful completion · case 05
A run with no remaining work is labeled completed despite having no final answer.
Workflow: workflow closes while follow-up work is queued · case 01
A run stops after its current turn despite queued follow-up items.
Workflow: workflow closes while follow-up work is queued · case 02
A run stops after its current turn despite queued follow-up items.
Workflow: workflow closes while follow-up work is queued · case 03
A run stops after its current turn despite queued follow-up items.
Workflow: workflow closes while follow-up work is queued · case 04
A run stops after its current turn despite queued follow-up items.
Workflow: workflow closes while follow-up work is queued · case 05
A run stops after its current turn despite queued follow-up items.
Workflow: only successful turns debit the workflow budget · case 01
Failed turns leave the allowance unchanged even though every completed turn consumes budget.
Workflow: only successful turns debit the workflow budget · case 02
Failed turns leave the allowance unchanged even though every completed turn consumes budget.
Workflow: only successful turns debit the workflow budget · case 03
Failed turns leave the allowance unchanged even though every completed turn consumes budget.
Workflow: only successful turns debit the workflow budget · case 04
Failed turns leave the allowance unchanged even though every completed turn consumes budget.
Workflow: only successful turns debit the workflow budget · case 05
Failed turns leave the allowance unchanged even though every completed turn consumes budget.
Workflow: duplicate result decrements pending count twice · case 01
A repeated result event makes the pending tool count negative.
Workflow: duplicate result decrements pending count twice · case 02
A repeated result event makes the pending tool count negative.
Workflow: duplicate result decrements pending count twice · case 03
A repeated result event makes the pending tool count negative.
Workflow: duplicate result decrements pending count twice · case 04
A repeated result event makes the pending tool count negative.
Workflow: duplicate result decrements pending count twice · case 05
A repeated result event makes the pending tool count negative.
Workflow: workflow bypasses its finalization barrier · case 01
The run closes before all required wrap-up checks finish.
Workflow: workflow bypasses its finalization barrier · case 02
The run closes before all required wrap-up checks finish.
Workflow: workflow bypasses its finalization barrier · case 03
The run closes before all required wrap-up checks finish.
Workflow: workflow bypasses its finalization barrier · case 04
The run closes before all required wrap-up checks finish.
Workflow: workflow bypasses its finalization barrier · case 05
The run closes before all required wrap-up checks finish.
Accuracy divides by predicted positives · case 01
Accuracy divides by predicted positives.
Accuracy divides by predicted positives · case 02
Accuracy divides by predicted positives.
Accuracy divides by predicted positives · case 03
Accuracy divides by predicted positives.
Accuracy divides by predicted positives · case 04
Accuracy divides by predicted positives.
Accuracy divides by predicted positives · case 05
Accuracy divides by predicted positives.
Confusion counts omit false negatives · case 01
Confusion counts omit false negatives.
Confusion counts omit false negatives · case 02
Confusion counts omit false negatives.
Confusion counts omit false negatives · case 03
Confusion counts omit false negatives.
Confusion counts omit false negatives · case 04
Confusion counts omit false negatives.
Confusion counts omit false negatives · case 05
Confusion counts omit false negatives.
Macro recall weights labels by support · case 01
Macro recall weights labels by support.
Macro recall weights labels by support · case 02
Macro recall weights labels by support.
Macro recall weights labels by support · case 03
Macro recall weights labels by support.
Macro recall weights labels by support · case 04
Macro recall weights labels by support.
Macro recall weights labels by support · case 05
Macro recall weights labels by support.
Micro precision omits false positives from the denominator · case 01
Micro precision omits false positives from the denominator.
Micro precision omits false positives from the denominator · case 02
Micro precision omits false positives from the denominator.
Micro precision omits false positives from the denominator · case 03
Micro precision omits false positives from the denominator.
Micro precision omits false positives from the denominator · case 04
Micro precision omits false positives from the denominator.
Micro precision omits false positives from the denominator · case 05
Micro precision omits false positives from the denominator.
Support weighted recall divides by label count · case 01
Support weighted recall divides by label count.
Support weighted recall divides by label count · case 02
Support weighted recall divides by label count.
Support weighted recall divides by label count · case 03
Support weighted recall divides by label count.
Support weighted recall divides by label count · case 04
Support weighted recall divides by label count.
Support weighted recall divides by label count · case 05
Support weighted recall divides by label count.
Macro F1 averages pooled counts instead of class scores · case 01
Macro F1 averages pooled counts instead of class scores.
Macro F1 averages pooled counts instead of class scores · case 02
Macro F1 averages pooled counts instead of class scores.
Macro F1 averages pooled counts instead of class scores · case 03
Macro F1 averages pooled counts instead of class scores.
Macro F1 averages pooled counts instead of class scores · case 04
Macro F1 averages pooled counts instead of class scores.
Macro F1 averages pooled counts instead of class scores · case 05
Macro F1 averages pooled counts instead of class scores.
Macro specificity is pooled across classes · case 01
Macro specificity is pooled across classes.
Macro specificity is pooled across classes · case 02
Macro specificity is pooled across classes.
Macro specificity is pooled across classes · case 03
Macro specificity is pooled across classes.
Macro specificity is pooled across classes · case 04
Macro specificity is pooled across classes.
Macro specificity is pooled across classes · case 05
Macro specificity is pooled across classes.
Balanced accuracy uses ordinary accuracy · case 01
Balanced accuracy uses ordinary accuracy.
Balanced accuracy uses ordinary accuracy · case 02
Balanced accuracy uses ordinary accuracy.
Balanced accuracy uses ordinary accuracy · case 03
Balanced accuracy uses ordinary accuracy.
Balanced accuracy uses ordinary accuracy · case 04
Balanced accuracy uses ordinary accuracy.
Balanced accuracy uses ordinary accuracy · case 05
Balanced accuracy uses ordinary accuracy.
Grouped recall pools groups before averaging · case 01
Grouped recall pools groups before averaging.
Grouped recall pools groups before averaging · case 02
Grouped recall pools groups before averaging.
Grouped recall pools groups before averaging · case 03
Grouped recall pools groups before averaging.
Grouped recall pools groups before averaging · case 04
Grouped recall pools groups before averaging.
Grouped recall pools groups before averaging · case 05
Grouped recall pools groups before averaging.
Grouped confusion counts average group matrices · case 01
Grouped confusion counts average group matrices.
Grouped confusion counts average group matrices · case 02
Grouped confusion counts average group matrices.
Grouped confusion counts average group matrices · case 03
Grouped confusion counts average group matrices.
Grouped confusion counts average group matrices · case 04
Grouped confusion counts average group matrices.
Grouped confusion counts average group matrices · case 05
Grouped confusion counts average group matrices.
Top k accuracy divides by candidate slots · case 01
Top k accuracy divides by candidate slots.
Top k accuracy divides by candidate slots · case 02
Top k accuracy divides by candidate slots.
Top k accuracy divides by candidate slots · case 03
Top k accuracy divides by candidate slots.
Top k accuracy divides by candidate slots · case 04
Top k accuracy divides by candidate slots.
Top k accuracy divides by candidate slots · case 05
Top k accuracy divides by candidate slots.
Multilabel subset accuracy counts matching labels · case 01
Multilabel subset accuracy counts matching labels.
Multilabel subset accuracy counts matching labels · case 02
Multilabel subset accuracy counts matching labels.
Multilabel subset accuracy counts matching labels · case 03
Multilabel subset accuracy counts matching labels.
Multilabel subset accuracy counts matching labels · case 04
Multilabel subset accuracy counts matching labels.
Multilabel subset accuracy counts matching labels · case 05
Multilabel subset accuracy counts matching labels.
INSPECTABLE BY DESIGN
Every result has a runnable source.
Runnable implementations with recorded outputs, source hashes, and explicit contracts. Related variants share a failure mechanism and belong together in evaluation splits.
Read the methodology ↗