Understand the failure.
Verify the repair.
Small, reproducible software failures. The broken implementation, the fix that didn’t work, and the one that passed—preserved together.
Explore the cases ↓How results are verified ↗WHAT THE ARCHIVE CONTAINS
100840 executable cases. 20168 are open.
Every case records the implementation that fails, the fix that did not work, and the repair that passed its checks—with recorded outputs and source hashes. This release adds 100840 cases across 20168 failure mechanisms and 254 domains.
The open tier gives you the failure and the unsuccessful fix for one case in every mechanism. The remaining 80672 cases, 5 variants per mechanism, are member-only: the verified repair, its recorded checks, and the full fixture suite are held in the member archive. Read the methodology ↗
A RECORD OF WHAT WENT WRONG
Browse the archive / 100840
Vector retrieval: A radius query includes points beyond the radius · case 01
A radius query includes points beyond the radius.
Vector retrieval: A radius query includes points beyond the radius · case 02
A radius query includes points beyond the radius.
Vector retrieval: A radius query includes points beyond the radius · case 03
A radius query includes points beyond the radius.
Vector retrieval: A radius query includes points beyond the radius · case 04
A radius query includes points beyond the radius.
Vector retrieval: A radius query includes points beyond the radius · case 05
A radius query includes points beyond the radius.
Vector retrieval: Rank fusion starts ranks at zero · case 01
Rank fusion starts ranks at zero.
Vector retrieval: Rank fusion starts ranks at zero · case 02
Rank fusion starts ranks at zero.
Vector retrieval: Rank fusion starts ranks at zero · case 03
Rank fusion starts ranks at zero.
Vector retrieval: Rank fusion starts ranks at zero · case 04
Rank fusion starts ranks at zero.
Vector retrieval: Rank fusion starts ranks at zero · case 05
Rank fusion starts ranks at zero.
Vector retrieval: A repeated source dominates every selected result · case 01
A repeated source dominates every selected result.
Vector retrieval: A repeated source dominates every selected result · case 02
A repeated source dominates every selected result.
Vector retrieval: A repeated source dominates every selected result · case 03
A repeated source dominates every selected result.
Vector retrieval: A repeated source dominates every selected result · case 04
A repeated source dominates every selected result.
Vector retrieval: A repeated source dominates every selected result · case 05
A repeated source dominates every selected result.
Vector retrieval: MMR repeatedly selects the top relevance score · case 01
MMR repeatedly selects the top relevance score.
Vector retrieval: MMR repeatedly selects the top relevance score · case 02
MMR repeatedly selects the top relevance score.
Vector retrieval: MMR repeatedly selects the top relevance score · case 03
MMR repeatedly selects the top relevance score.
Vector retrieval: MMR repeatedly selects the top relevance score · case 04
MMR repeatedly selects the top relevance score.
Vector retrieval: MMR repeatedly selects the top relevance score · case 05
MMR repeatedly selects the top relevance score.
Vector retrieval: Per-shard top results are concatenated without global ranking · case 01
Per-shard top results are concatenated without global ranking.
Vector retrieval: Per-shard top results are concatenated without global ranking · case 02
Per-shard top results are concatenated without global ranking.
Vector retrieval: Per-shard top results are concatenated without global ranking · case 03
Per-shard top results are concatenated without global ranking.
Vector retrieval: Per-shard top results are concatenated without global ranking · case 04
Per-shard top results are concatenated without global ranking.
Vector retrieval: Per-shard top results are concatenated without global ranking · case 05
Per-shard top results are concatenated without global ranking.
Vector retrieval: Duplicate IDs keep the first shard score · case 01
Duplicate IDs keep the first shard score.
Vector retrieval: Duplicate IDs keep the first shard score · case 02
Duplicate IDs keep the first shard score.
Vector retrieval: Duplicate IDs keep the first shard score · case 03
Duplicate IDs keep the first shard score.
Vector retrieval: Duplicate IDs keep the first shard score · case 04
Duplicate IDs keep the first shard score.
Vector retrieval: Duplicate IDs keep the first shard score · case 05
Duplicate IDs keep the first shard score.
Vector retrieval: Scores are paired with the wrong candidate IDs · case 01
Scores are paired with the wrong candidate IDs.
Vector retrieval: Scores are paired with the wrong candidate IDs · case 02
Scores are paired with the wrong candidate IDs.
Vector retrieval: Scores are paired with the wrong candidate IDs · case 03
Scores are paired with the wrong candidate IDs.
Vector retrieval: Scores are paired with the wrong candidate IDs · case 04
Scores are paired with the wrong candidate IDs.
Vector retrieval: Scores are paired with the wrong candidate IDs · case 05
Scores are paired with the wrong candidate IDs.
Vector retrieval: A query vector retrieves its own record · case 01
A query vector retrieves its own record.
Vector retrieval: A query vector retrieves its own record · case 02
A query vector retrieves its own record.
Vector retrieval: A query vector retrieves its own record · case 03
A query vector retrieves its own record.
Vector retrieval: A query vector retrieves its own record · case 04
A query vector retrieves its own record.
Vector retrieval: A query vector retrieves its own record · case 05
A query vector retrieves its own record.
RAG assembly stops at the first over-budget chunk · case 01
RAG assembly stops at the first over-budget chunk.
RAG assembly stops at the first over-budget chunk · case 02
RAG assembly stops at the first over-budget chunk.
RAG assembly stops at the first over-budget chunk · case 03
RAG assembly stops at the first over-budget chunk.
RAG assembly stops at the first over-budget chunk · case 04
RAG assembly stops at the first over-budget chunk.
RAG assembly stops at the first over-budget chunk · case 05
RAG assembly stops at the first over-budget chunk.
A per-source limit lets one source contribute too many chunks · case 01
A per-source limit lets one source contribute too many chunks.
A per-source limit lets one source contribute too many chunks · case 02
A per-source limit lets one source contribute too many chunks.
A per-source limit lets one source contribute too many chunks · case 03
A per-source limit lets one source contribute too many chunks.
A per-source limit lets one source contribute too many chunks · case 04
A per-source limit lets one source contribute too many chunks.
A per-source limit lets one source contribute too many chunks · case 05
A per-source limit lets one source contribute too many chunks.
Context assembly omits separator tokens from its budget · case 01
Context assembly omits separator tokens from its budget.
Context assembly omits separator tokens from its budget · case 02
Context assembly omits separator tokens from its budget.
Context assembly omits separator tokens from its budget · case 03
Context assembly omits separator tokens from its budget.
Context assembly omits separator tokens from its budget · case 04
Context assembly omits separator tokens from its budget.
Context assembly omits separator tokens from its budget · case 05
Context assembly omits separator tokens from its budget.
Overlapping neighboring chunks repeat evidence tokens · case 01
Overlapping neighboring chunks repeat evidence tokens.
Overlapping neighboring chunks repeat evidence tokens · case 02
Overlapping neighboring chunks repeat evidence tokens.
Overlapping neighboring chunks repeat evidence tokens · case 03
Overlapping neighboring chunks repeat evidence tokens.
Overlapping neighboring chunks repeat evidence tokens · case 04
Overlapping neighboring chunks repeat evidence tokens.
Overlapping neighboring chunks repeat evidence tokens · case 05
Overlapping neighboring chunks repeat evidence tokens.
Context coverage interleaves document chunks without breaking prefixes · case 01
Context coverage interleaves document chunks without breaking prefixes.
Context coverage interleaves document chunks without breaking prefixes · case 02
Context coverage interleaves document chunks without breaking prefixes.
Context coverage interleaves document chunks without breaking prefixes · case 03
Context coverage interleaves document chunks without breaking prefixes.
Context coverage interleaves document chunks without breaking prefixes · case 04
Context coverage interleaves document chunks without breaking prefixes.
Context coverage interleaves document chunks without breaking prefixes · case 05
Context coverage interleaves document chunks without breaking prefixes.
Whole-chunk prefix assembly slices text at a token limit · case 01
Whole-chunk prefix assembly slices text at a token limit.
Whole-chunk prefix assembly slices text at a token limit · case 02
Whole-chunk prefix assembly slices text at a token limit.
Whole-chunk prefix assembly slices text at a token limit · case 03
Whole-chunk prefix assembly slices text at a token limit.
Whole-chunk prefix assembly slices text at a token limit · case 04
Whole-chunk prefix assembly slices text at a token limit.
Whole-chunk prefix assembly slices text at a token limit · case 05
Whole-chunk prefix assembly slices text at a token limit.
Rank-ordered evidence crowds out other sources · case 01
Rank-ordered evidence crowds out other sources.
Rank-ordered evidence crowds out other sources · case 02
Rank-ordered evidence crowds out other sources.
Rank-ordered evidence crowds out other sources · case 03
Rank-ordered evidence crowds out other sources.
Rank-ordered evidence crowds out other sources · case 04
Rank-ordered evidence crowds out other sources.
Rank-ordered evidence crowds out other sources · case 05
Rank-ordered evidence crowds out other sources.
A grouped evidence bundle is partially included · case 01
A grouped evidence bundle is partially included.
A grouped evidence bundle is partially included · case 02
A grouped evidence bundle is partially included.
A grouped evidence bundle is partially included · case 03
A grouped evidence bundle is partially included.
A grouped evidence bundle is partially included · case 04
A grouped evidence bundle is partially included.
A grouped evidence bundle is partially included · case 05
A grouped evidence bundle is partially included.
Context budget double-counts shared token overlap · case 01
Context budget double-counts shared token overlap.
Context budget double-counts shared token overlap · case 02
Context budget double-counts shared token overlap.
Context budget double-counts shared token overlap · case 03
Context budget double-counts shared token overlap.
Context budget double-counts shared token overlap · case 04
Context budget double-counts shared token overlap.
Context budget double-counts shared token overlap · case 05
Context budget double-counts shared token overlap.
The assembler fills budget without required source coverage · case 01
The assembler fills budget without required source coverage.
The assembler fills budget without required source coverage · case 02
The assembler fills budget without required source coverage.
The assembler fills budget without required source coverage · case 03
The assembler fills budget without required source coverage.
The assembler fills budget without required source coverage · case 04
The assembler fills budget without required source coverage.
The assembler fills budget without required source coverage · case 05
The assembler fills budget without required source coverage.
Context is built from earliest rather than newest evidence · case 01
Context is built from earliest rather than newest evidence.
Context is built from earliest rather than newest evidence · case 02
Context is built from earliest rather than newest evidence.
Context is built from earliest rather than newest evidence · case 03
Context is built from earliest rather than newest evidence.
Context is built from earliest rather than newest evidence · case 04
Context is built from earliest rather than newest evidence.
Context is built from earliest rather than newest evidence · case 05
Context is built from earliest rather than newest evidence.
Neighboring context expands on the wrong side of an anchor · case 01
Neighboring context expands on the wrong side of an anchor.
Neighboring context expands on the wrong side of an anchor · case 02
Neighboring context expands on the wrong side of an anchor.
Neighboring context expands on the wrong side of an anchor · case 03
Neighboring context expands on the wrong side of an anchor.
Neighboring context expands on the wrong side of an anchor · case 04
Neighboring context expands on the wrong side of an anchor.
Neighboring context expands on the wrong side of an anchor · case 05
Neighboring context expands on the wrong side of an anchor.
INSPECTABLE BY DESIGN
Every result has a runnable source.
Runnable implementations with recorded outputs, source hashes, and explicit contracts. Related variants share a failure mechanism and belong together in evaluation splits.
Read the methodology ↗