Understand the failure.
Verify the repair.
Small, reproducible software failures. The broken implementation, the fix that didn’t work, and the one that passed—preserved together.
Explore the cases ↓How results are verified ↗WHAT THE ARCHIVE CONTAINS
100840 executable cases. 20168 are open.
Every case records the implementation that fails, the fix that did not work, and the repair that passed its checks—with recorded outputs and source hashes. This release adds 100840 cases across 20168 failure mechanisms and 254 domains.
The open tier gives you the failure and the unsuccessful fix for one case in every mechanism. The remaining 80672 cases, 5 variants per mechanism, are member-only: the verified repair, its recorded checks, and the full fixture suite are held in the member archive. Read the methodology ↗
A RECORD OF WHAT WENT WRONG
Browse the archive / 100840
Preference pairs: zero-confidence annotations count as preferences · case 01
zero-confidence annotations count as preferences.
Preference pairs: zero-confidence annotations count as preferences · case 02
zero-confidence annotations count as preferences.
Preference pairs: zero-confidence annotations count as preferences · case 03
zero-confidence annotations count as preferences.
Preference pairs: zero-confidence annotations count as preferences · case 04
zero-confidence annotations count as preferences.
Preference pairs: zero-confidence annotations count as preferences · case 05
zero-confidence annotations count as preferences.
Preference pairs: ordinal labels are treated as numeric distances · case 01
ordinal labels are treated as numeric distances.
Preference pairs: ordinal labels are treated as numeric distances · case 02
ordinal labels are treated as numeric distances.
Preference pairs: ordinal labels are treated as numeric distances · case 03
ordinal labels are treated as numeric distances.
Preference pairs: ordinal labels are treated as numeric distances · case 04
ordinal labels are treated as numeric distances.
Preference pairs: ordinal labels are treated as numeric distances · case 05
ordinal labels are treated as numeric distances.
Preference pairs: stale annotation revisions remain active · case 01
stale annotation revisions remain active.
Preference pairs: stale annotation revisions remain active · case 02
stale annotation revisions remain active.
Preference pairs: stale annotation revisions remain active · case 03
stale annotation revisions remain active.
Preference pairs: stale annotation revisions remain active · case 04
stale annotation revisions remain active.
Preference pairs: stale annotation revisions remain active · case 05
stale annotation revisions remain active.
Preference pairs: pair generation favors one orientation · case 01
pair generation favors one orientation.
Preference pairs: pair generation favors one orientation · case 02
pair generation favors one orientation.
Preference pairs: pair generation favors one orientation · case 03
pair generation favors one orientation.
Preference pairs: pair generation favors one orientation · case 04
pair generation favors one orientation.
Preference pairs: pair generation favors one orientation · case 05
pair generation favors one orientation.
Preference pairs: disagreeing annotations are collapsed to a winner · case 01
disagreeing annotations are collapsed to a winner.
Preference pairs: disagreeing annotations are collapsed to a winner · case 02
disagreeing annotations are collapsed to a winner.
Preference pairs: disagreeing annotations are collapsed to a winner · case 03
disagreeing annotations are collapsed to a winner.
Preference pairs: disagreeing annotations are collapsed to a winner · case 04
disagreeing annotations are collapsed to a winner.
Preference pairs: disagreeing annotations are collapsed to a winner · case 05
disagreeing annotations are collapsed to a winner.
Preference pairs: responses from different sources are paired · case 01
responses from different sources are paired.
Preference pairs: responses from different sources are paired · case 02
responses from different sources are paired.
Preference pairs: responses from different sources are paired · case 03
responses from different sources are paired.
Preference pairs: responses from different sources are paired · case 04
responses from different sources are paired.
Preference pairs: responses from different sources are paired · case 05
responses from different sources are paired.
Preference pairs: pair sampling drops minority strata · case 01
pair sampling drops minority strata.
Preference pairs: pair sampling drops minority strata · case 02
pair sampling drops minority strata.
Preference pairs: pair sampling drops minority strata · case 03
pair sampling drops minority strata.
Preference pairs: pair sampling drops minority strata · case 04
pair sampling drops minority strata.
Preference pairs: pair sampling drops minority strata · case 05
pair sampling drops minority strata.
Preference pairs: pair text budget is consumed by the first endpoint · case 01
pair text budget is consumed by the first endpoint.
Preference pairs: pair text budget is consumed by the first endpoint · case 02
pair text budget is consumed by the first endpoint.
Preference pairs: pair text budget is consumed by the first endpoint · case 03
pair text budget is consumed by the first endpoint.
Preference pairs: pair text budget is consumed by the first endpoint · case 04
pair text budget is consumed by the first endpoint.
Preference pairs: pair text budget is consumed by the first endpoint · case 05
pair text budget is consumed by the first endpoint.
Insertion shifts spans at and after the insertion point · case 01
Insertion shifts spans at and after the insertion point.
Insertion shifts spans at and after the insertion point · case 02
Insertion shifts spans at and after the insertion point.
Insertion shifts spans at and after the insertion point · case 03
Insertion shifts spans at and after the insertion point.
Insertion shifts spans at and after the insertion point · case 04
Insertion shifts spans at and after the insertion point.
Insertion shifts spans at and after the insertion point · case 05
Insertion shifts spans at and after the insertion point.
Deleted text clips spans to the surviving boundary · case 01
Deleted text clips spans to the surviving boundary.
Deleted text clips spans to the surviving boundary · case 02
Deleted text clips spans to the surviving boundary.
Deleted text clips spans to the surviving boundary · case 03
Deleted text clips spans to the surviving boundary.
Deleted text clips spans to the surviving boundary · case 04
Deleted text clips spans to the surviving boundary.
Deleted text clips spans to the surviving boundary · case 05
Deleted text clips spans to the surviving boundary.
Replacement length delta stretches later annotations · case 01
Replacement length delta stretches later annotations.
Replacement length delta stretches later annotations · case 02
Replacement length delta stretches later annotations.
Replacement length delta stretches later annotations · case 03
Replacement length delta stretches later annotations.
Replacement length delta stretches later annotations · case 04
Replacement length delta stretches later annotations.
Replacement length delta stretches later annotations · case 05
Replacement length delta stretches later annotations.
Token index maps to the wrong character boundary · case 01
Token index maps to the wrong character boundary.
Token index maps to the wrong character boundary · case 02
Token index maps to the wrong character boundary.
Token index maps to the wrong character boundary · case 03
Token index maps to the wrong character boundary.
Token index maps to the wrong character boundary · case 04
Token index maps to the wrong character boundary.
Token index maps to the wrong character boundary · case 05
Token index maps to the wrong character boundary.
Character span misses a token that it partially covers · case 01
Character span misses a token that it partially covers.
Character span misses a token that it partially covers · case 02
Character span misses a token that it partially covers.
Character span misses a token that it partially covers · case 03
Character span misses a token that it partially covers.
Character span misses a token that it partially covers · case 04
Character span misses a token that it partially covers.
Character span misses a token that it partially covers · case 05
Character span misses a token that it partially covers.
Token labels leak across neighboring annotations · case 01
Token labels leak across neighboring annotations.
Token labels leak across neighboring annotations · case 02
Token labels leak across neighboring annotations.
Token labels leak across neighboring annotations · case 03
Token labels leak across neighboring annotations.
Token labels leak across neighboring annotations · case 04
Token labels leak across neighboring annotations.
Token labels leak across neighboring annotations · case 05
Token labels leak across neighboring annotations.
Adjacent same-label spans remain fragmented · case 01
Adjacent same-label spans remain fragmented.
Adjacent same-label spans remain fragmented · case 02
Adjacent same-label spans remain fragmented.
Adjacent same-label spans remain fragmented · case 03
Adjacent same-label spans remain fragmented.
Adjacent same-label spans remain fragmented · case 04
Adjacent same-label spans remain fragmented.
Adjacent same-label spans remain fragmented · case 05
Adjacent same-label spans remain fragmented.
Nested annotations are sorted by end before start · case 01
Nested annotations are sorted by end before start.
Nested annotations are sorted by end before start · case 02
Nested annotations are sorted by end before start.
Nested annotations are sorted by end before start · case 03
Nested annotations are sorted by end before start.
Nested annotations are sorted by end before start · case 04
Nested annotations are sorted by end before start.
Nested annotations are sorted by end before start · case 05
Nested annotations are sorted by end before start.
Overlapping annotation priority is ignored · case 01
Overlapping annotation priority is ignored.
Overlapping annotation priority is ignored · case 02
Overlapping annotation priority is ignored.
Overlapping annotation priority is ignored · case 03
Overlapping annotation priority is ignored.
Overlapping annotation priority is ignored · case 04
Overlapping annotation priority is ignored.
Overlapping annotation priority is ignored · case 05
Overlapping annotation priority is ignored.
Touching spans are incorrectly reported as overlapping · case 01
Touching spans are incorrectly reported as overlapping.
Touching spans are incorrectly reported as overlapping · case 02
Touching spans are incorrectly reported as overlapping.
Touching spans are incorrectly reported as overlapping · case 03
Touching spans are incorrectly reported as overlapping.
Touching spans are incorrectly reported as overlapping · case 04
Touching spans are incorrectly reported as overlapping.
Touching spans are incorrectly reported as overlapping · case 05
Touching spans are incorrectly reported as overlapping.
Windowed export keeps original coordinates after clipping · case 01
Windowed export keeps original coordinates after clipping.
Windowed export keeps original coordinates after clipping · case 02
Windowed export keeps original coordinates after clipping.
Windowed export keeps original coordinates after clipping · case 03
Windowed export keeps original coordinates after clipping.
Windowed export keeps original coordinates after clipping · case 04
Windowed export keeps original coordinates after clipping.
Windowed export keeps original coordinates after clipping · case 05
Windowed export keeps original coordinates after clipping.
Boundary-touching spans pass a containment filter · case 01
Boundary-touching spans pass a containment filter.
Boundary-touching spans pass a containment filter · case 02
Boundary-touching spans pass a containment filter.
Boundary-touching spans pass a containment filter · case 03
Boundary-touching spans pass a containment filter.
Boundary-touching spans pass a containment filter · case 04
Boundary-touching spans pass a containment filter.
Boundary-touching spans pass a containment filter · case 05
Boundary-touching spans pass a containment filter.
INSPECTABLE BY DESIGN
Every result has a runnable source.
Runnable implementations with recorded outputs, source hashes, and explicit contracts. Related variants share a failure mechanism and belong together in evaluation splits.
Read the methodology ↗