FAILURE MAP

Understand the failure.
Verify the repair.

Small, reproducible software failures. The broken implementation, the fix that didn’t work, and the one that passed—preserved together.

Explore the cases ↓How results are verified ↗
100840Executable case variants
20168Distinct failure mechanisms
302520Executed implementations
20168Open-access cases

WHAT THE ARCHIVE CONTAINS

100840 executable cases. 20168 are open.

Every case records the implementation that fails, the fix that did not work, and the repair that passed its checks—with recorded outputs and source hashes. This release adds 100840 cases across 20168 failure mechanisms and 254 domains.

The open tier gives you the failure and the unsuccessful fix for one case in every mechanism. The remaining 80672 cases, 5 variants per mechanism, are member-only: the verified repair, its recorded checks, and the full fixture suite are held in the member archive. Read the methodology ↗

A RECORD OF WHAT WENT WRONG

Browse the archive / 100840

Python · Standard library
REFERENCEFAILURE MECHANISMDOMAINACCESS
FA-27001

A client serializes a typed range request plan: suffix counts are introduced by a leading dash · case 01

A client serializes a typed range request plan: suffix counts are introduced by a leading dash.

HTTP ranges● Open access↗
FA-27002

A client serializes a typed range request plan: suffix counts are introduced by a leading dash · case 02

A client serializes a typed range request plan: suffix counts are introduced by a leading dash.

HTTP ranges◈ Members↗
FA-27003

A client serializes a typed range request plan: suffix counts are introduced by a leading dash · case 03

A client serializes a typed range request plan: suffix counts are introduced by a leading dash.

HTTP ranges◈ Members↗
FA-27004

A client serializes a typed range request plan: suffix counts are introduced by a leading dash · case 04

A client serializes a typed range request plan: suffix counts are introduced by a leading dash.

HTTP ranges◈ Members↗
FA-27005

A client serializes a typed range request plan: suffix counts are introduced by a leading dash · case 05

A client serializes a typed range request plan: suffix counts are introduced by a leading dash.

HTTP ranges◈ Members↗
FA-27006

A client serializes a typed range request plan: outbound members preserve caller priority order · case 01

A client serializes a typed range request plan: outbound members preserve caller priority order.

HTTP ranges● Open access↗
FA-27007

A client serializes a typed range request plan: outbound members preserve caller priority order · case 02

A client serializes a typed range request plan: outbound members preserve caller priority order.

HTTP ranges◈ Members↗
FA-27008

A client serializes a typed range request plan: outbound members preserve caller priority order · case 03

A client serializes a typed range request plan: outbound members preserve caller priority order.

HTTP ranges◈ Members↗
FA-27009

A client serializes a typed range request plan: outbound members preserve caller priority order · case 04

A client serializes a typed range request plan: outbound members preserve caller priority order.

HTTP ranges◈ Members↗
FA-27010

A client serializes a typed range request plan: outbound members preserve caller priority order · case 05

A client serializes a typed range request plan: outbound members preserve caller priority order.

HTTP ranges◈ Members↗
FA-27011

A client serializes a typed range request plan: multiple members share a single comma-separated field · case 01

A client serializes a typed range request plan: multiple members share a single comma-separated field.

HTTP ranges● Open access↗
FA-27012

A client serializes a typed range request plan: multiple members share a single comma-separated field · case 02

A client serializes a typed range request plan: multiple members share a single comma-separated field.

HTTP ranges◈ Members↗
FA-27013

A client serializes a typed range request plan: multiple members share a single comma-separated field · case 03

A client serializes a typed range request plan: multiple members share a single comma-separated field.

HTTP ranges◈ Members↗
FA-27014

A client serializes a typed range request plan: multiple members share a single comma-separated field · case 04

A client serializes a typed range request plan: multiple members share a single comma-separated field.

HTTP ranges◈ Members↗
FA-27015

A client serializes a typed range request plan: multiple members share a single comma-separated field · case 05

A client serializes a typed range request plan: multiple members share a single comma-separated field.

HTTP ranges◈ Members↗
FA-27016

A client serializes a typed range request plan: Range requests do not send response-only Content-Range fields · case 01

A client serializes a typed range request plan: Range requests do not send response-only Content-Range fields.

HTTP ranges● Open access↗
FA-27017

A client serializes a typed range request plan: Range requests do not send response-only Content-Range fields · case 02

A client serializes a typed range request plan: Range requests do not send response-only Content-Range fields.

HTTP ranges◈ Members↗
FA-27018

A client serializes a typed range request plan: Range requests do not send response-only Content-Range fields · case 03

A client serializes a typed range request plan: Range requests do not send response-only Content-Range fields.

HTTP ranges◈ Members↗
FA-27019

A client serializes a typed range request plan: Range requests do not send response-only Content-Range fields · case 04

A client serializes a typed range request plan: Range requests do not send response-only Content-Range fields.

HTTP ranges◈ Members↗
FA-27020

A client serializes a typed range request plan: Range requests do not send response-only Content-Range fields · case 05

A client serializes a typed range request plan: Range requests do not send response-only Content-Range fields.

HTTP ranges◈ Members↗
FA-27021

A client serializes a typed range request plan: an empty selector plan emits no conditional range field · case 01

A client serializes a typed range request plan: an empty selector plan emits no conditional range field.

HTTP ranges● Open access↗
FA-27022

A client serializes a typed range request plan: an empty selector plan emits no conditional range field · case 02

A client serializes a typed range request plan: an empty selector plan emits no conditional range field.

HTTP ranges◈ Members↗
FA-27023

A client serializes a typed range request plan: an empty selector plan emits no conditional range field · case 03

A client serializes a typed range request plan: an empty selector plan emits no conditional range field.

HTTP ranges◈ Members↗
FA-27024

A client serializes a typed range request plan: an empty selector plan emits no conditional range field · case 04

A client serializes a typed range request plan: an empty selector plan emits no conditional range field.

HTTP ranges◈ Members↗
FA-27025

A client serializes a typed range request plan: an empty selector plan emits no conditional range field · case 05

A client serializes a typed range request plan: an empty selector plan emits no conditional range field.

HTTP ranges◈ Members↗
FA-27026

A client serializes a typed range request plan: entity validators retain their quoted wire form · case 01

A client serializes a typed range request plan: entity validators retain their quoted wire form.

HTTP ranges● Open access↗
FA-27027

A client serializes a typed range request plan: entity validators retain their quoted wire form · case 02

A client serializes a typed range request plan: entity validators retain their quoted wire form.

HTTP ranges◈ Members↗
FA-27028

A client serializes a typed range request plan: entity validators retain their quoted wire form · case 03

A client serializes a typed range request plan: entity validators retain their quoted wire form.

HTTP ranges◈ Members↗
FA-27029

A client serializes a typed range request plan: entity validators retain their quoted wire form · case 04

A client serializes a typed range request plan: entity validators retain their quoted wire form.

HTTP ranges◈ Members↗
FA-27030

A client serializes a typed range request plan: entity validators retain their quoted wire form · case 05

A client serializes a typed range request plan: entity validators retain their quoted wire form.

HTTP ranges◈ Members↗
FA-27031

A client serializes a typed range request plan: date validators are not serialized as opaque entity tags · case 01

A client serializes a typed range request plan: date validators are not serialized as opaque entity tags.

HTTP ranges● Open access↗
FA-27032

A client serializes a typed range request plan: date validators are not serialized as opaque entity tags · case 02

A client serializes a typed range request plan: date validators are not serialized as opaque entity tags.

HTTP ranges◈ Members↗
FA-27033

A client serializes a typed range request plan: date validators are not serialized as opaque entity tags · case 03

A client serializes a typed range request plan: date validators are not serialized as opaque entity tags.

HTTP ranges◈ Members↗
FA-27034

A client serializes a typed range request plan: date validators are not serialized as opaque entity tags · case 04

A client serializes a typed range request plan: date validators are not serialized as opaque entity tags.

HTTP ranges◈ Members↗
FA-27035

A client serializes a typed range request plan: date validators are not serialized as opaque entity tags · case 05

A client serializes a typed range request plan: date validators are not serialized as opaque entity tags.

HTTP ranges◈ Members↗
FA-27036

A client serializes a typed range request plan: byte-stable downloads request identity when configured · case 01

A client serializes a typed range request plan: byte-stable downloads request identity when configured.

HTTP ranges● Open access↗
FA-27037

A client serializes a typed range request plan: byte-stable downloads request identity when configured · case 02

A client serializes a typed range request plan: byte-stable downloads request identity when configured.

HTTP ranges◈ Members↗
FA-27038

A client serializes a typed range request plan: byte-stable downloads request identity when configured · case 03

A client serializes a typed range request plan: byte-stable downloads request identity when configured.

HTTP ranges◈ Members↗
FA-27039

A client serializes a typed range request plan: byte-stable downloads request identity when configured · case 04

A client serializes a typed range request plan: byte-stable downloads request identity when configured.

HTTP ranges◈ Members↗
FA-27040

A client serializes a typed range request plan: byte-stable downloads request identity when configured · case 05

A client serializes a typed range request plan: byte-stable downloads request identity when configured.

HTTP ranges◈ Members↗
FA-27041

A proxy coalesces nearby requested spans into economical origin reads: unsorted wire order must not drive origin coalescing · case 01

A proxy coalesces nearby requested spans into economical origin reads: unsorted wire order must not drive origin coalescing.

HTTP ranges● Open access↗
FA-27042

A proxy coalesces nearby requested spans into economical origin reads: unsorted wire order must not drive origin coalescing · case 02

A proxy coalesces nearby requested spans into economical origin reads: unsorted wire order must not drive origin coalescing.

HTTP ranges◈ Members↗
FA-27043

A proxy coalesces nearby requested spans into economical origin reads: unsorted wire order must not drive origin coalescing · case 03

A proxy coalesces nearby requested spans into economical origin reads: unsorted wire order must not drive origin coalescing.

HTTP ranges◈ Members↗
FA-27044

A proxy coalesces nearby requested spans into economical origin reads: unsorted wire order must not drive origin coalescing · case 04

A proxy coalesces nearby requested spans into economical origin reads: unsorted wire order must not drive origin coalescing.

HTTP ranges◈ Members↗
FA-27045

A proxy coalesces nearby requested spans into economical origin reads: unsorted wire order must not drive origin coalescing · case 05

A proxy coalesces nearby requested spans into economical origin reads: unsorted wire order must not drive origin coalescing.

HTTP ranges◈ Members↗
FA-27046

A proxy coalesces nearby requested spans into economical origin reads: empty logical selectors must not allocate backend fetches · case 01

A proxy coalesces nearby requested spans into economical origin reads: empty logical selectors must not allocate backend fetches.

HTTP ranges● Open access↗
FA-27047

A proxy coalesces nearby requested spans into economical origin reads: empty logical selectors must not allocate backend fetches · case 02

A proxy coalesces nearby requested spans into economical origin reads: empty logical selectors must not allocate backend fetches.

HTTP ranges◈ Members↗
FA-27048

A proxy coalesces nearby requested spans into economical origin reads: empty logical selectors must not allocate backend fetches · case 03

A proxy coalesces nearby requested spans into economical origin reads: empty logical selectors must not allocate backend fetches.

HTTP ranges◈ Members↗
FA-27049

A proxy coalesces nearby requested spans into economical origin reads: empty logical selectors must not allocate backend fetches · case 04

A proxy coalesces nearby requested spans into economical origin reads: empty logical selectors must not allocate backend fetches.

HTTP ranges◈ Members↗
FA-27050

A proxy coalesces nearby requested spans into economical origin reads: empty logical selectors must not allocate backend fetches · case 05

A proxy coalesces nearby requested spans into economical origin reads: empty logical selectors must not allocate backend fetches.

HTTP ranges◈ Members↗
FA-27051

A proxy coalesces nearby requested spans into economical origin reads: reads at the configured merge gap remain eligible · case 01

A proxy coalesces nearby requested spans into economical origin reads: reads at the configured merge gap remain eligible.

HTTP ranges● Open access↗
FA-27052

A proxy coalesces nearby requested spans into economical origin reads: reads at the configured merge gap remain eligible · case 02

A proxy coalesces nearby requested spans into economical origin reads: reads at the configured merge gap remain eligible.

HTTP ranges◈ Members↗
FA-27053

A proxy coalesces nearby requested spans into economical origin reads: reads at the configured merge gap remain eligible · case 03

A proxy coalesces nearby requested spans into economical origin reads: reads at the configured merge gap remain eligible.

HTTP ranges◈ Members↗
FA-27054

A proxy coalesces nearby requested spans into economical origin reads: reads at the configured merge gap remain eligible · case 04

A proxy coalesces nearby requested spans into economical origin reads: reads at the configured merge gap remain eligible.

HTTP ranges◈ Members↗
FA-27055

A proxy coalesces nearby requested spans into economical origin reads: reads at the configured merge gap remain eligible · case 05

A proxy coalesces nearby requested spans into economical origin reads: reads at the configured merge gap remain eligible.

HTTP ranges◈ Members↗
FA-27056

A proxy coalesces nearby requested spans into economical origin reads: coalescing cannot fetch forbidden bytes absent from requested payloads · case 01

A proxy coalesces nearby requested spans into economical origin reads: coalescing cannot fetch forbidden bytes absent from requested payloads.

HTTP ranges● Open access↗
FA-27057

A proxy coalesces nearby requested spans into economical origin reads: coalescing cannot fetch forbidden bytes absent from requested payloads · case 02

A proxy coalesces nearby requested spans into economical origin reads: coalescing cannot fetch forbidden bytes absent from requested payloads.

HTTP ranges◈ Members↗
FA-27058

A proxy coalesces nearby requested spans into economical origin reads: coalescing cannot fetch forbidden bytes absent from requested payloads · case 03

A proxy coalesces nearby requested spans into economical origin reads: coalescing cannot fetch forbidden bytes absent from requested payloads.

HTTP ranges◈ Members↗
FA-27059

A proxy coalesces nearby requested spans into economical origin reads: coalescing cannot fetch forbidden bytes absent from requested payloads · case 04

A proxy coalesces nearby requested spans into economical origin reads: coalescing cannot fetch forbidden bytes absent from requested payloads.

HTTP ranges◈ Members↗
FA-27060

A proxy coalesces nearby requested spans into economical origin reads: coalescing cannot fetch forbidden bytes absent from requested payloads · case 05

A proxy coalesces nearby requested spans into economical origin reads: coalescing cannot fetch forbidden bytes absent from requested payloads.

HTTP ranges◈ Members↗
FA-27061

A proxy coalesces nearby requested spans into economical origin reads: contained requests must not shrink a larger origin read · case 01

A proxy coalesces nearby requested spans into economical origin reads: contained requests must not shrink a larger origin read.

HTTP ranges● Open access↗
FA-27062

A proxy coalesces nearby requested spans into economical origin reads: contained requests must not shrink a larger origin read · case 02

A proxy coalesces nearby requested spans into economical origin reads: contained requests must not shrink a larger origin read.

HTTP ranges◈ Members↗
FA-27063

A proxy coalesces nearby requested spans into economical origin reads: contained requests must not shrink a larger origin read · case 03

A proxy coalesces nearby requested spans into economical origin reads: contained requests must not shrink a larger origin read.

HTTP ranges◈ Members↗
FA-27064

A proxy coalesces nearby requested spans into economical origin reads: contained requests must not shrink a larger origin read · case 04

A proxy coalesces nearby requested spans into economical origin reads: contained requests must not shrink a larger origin read.

HTTP ranges◈ Members↗
FA-27065

A proxy coalesces nearby requested spans into economical origin reads: contained requests must not shrink a larger origin read · case 05

A proxy coalesces nearby requested spans into economical origin reads: contained requests must not shrink a larger origin read.

HTTP ranges◈ Members↗
FA-27066

A proxy coalesces nearby requested spans into economical origin reads: new fetches map to their newly allocated index · case 01

A proxy coalesces nearby requested spans into economical origin reads: new fetches map to their newly allocated index.

HTTP ranges● Open access↗
FA-27067

A proxy coalesces nearby requested spans into economical origin reads: new fetches map to their newly allocated index · case 02

A proxy coalesces nearby requested spans into economical origin reads: new fetches map to their newly allocated index.

HTTP ranges◈ Members↗
FA-27068

A proxy coalesces nearby requested spans into economical origin reads: new fetches map to their newly allocated index · case 03

A proxy coalesces nearby requested spans into economical origin reads: new fetches map to their newly allocated index.

HTTP ranges◈ Members↗
FA-27069

A proxy coalesces nearby requested spans into economical origin reads: new fetches map to their newly allocated index · case 04

A proxy coalesces nearby requested spans into economical origin reads: new fetches map to their newly allocated index.

HTTP ranges◈ Members↗
FA-27070

A proxy coalesces nearby requested spans into economical origin reads: new fetches map to their newly allocated index · case 05

A proxy coalesces nearby requested spans into economical origin reads: new fetches map to their newly allocated index.

HTTP ranges◈ Members↗
FA-27071

A proxy coalesces nearby requested spans into economical origin reads: fetch mappings are returned in caller order after read sorting · case 01

A proxy coalesces nearby requested spans into economical origin reads: fetch mappings are returned in caller order after read sorting.

HTTP ranges● Open access↗
FA-27072

A proxy coalesces nearby requested spans into economical origin reads: fetch mappings are returned in caller order after read sorting · case 02

A proxy coalesces nearby requested spans into economical origin reads: fetch mappings are returned in caller order after read sorting.

HTTP ranges◈ Members↗
FA-27073

A proxy coalesces nearby requested spans into economical origin reads: fetch mappings are returned in caller order after read sorting · case 03

A proxy coalesces nearby requested spans into economical origin reads: fetch mappings are returned in caller order after read sorting.

HTTP ranges◈ Members↗
FA-27074

A proxy coalesces nearby requested spans into economical origin reads: fetch mappings are returned in caller order after read sorting · case 04

A proxy coalesces nearby requested spans into economical origin reads: fetch mappings are returned in caller order after read sorting.

HTTP ranges◈ Members↗
FA-27075

A proxy coalesces nearby requested spans into economical origin reads: fetch mappings are returned in caller order after read sorting · case 05

A proxy coalesces nearby requested spans into economical origin reads: fetch mappings are returned in caller order after read sorting.

HTTP ranges◈ Members↗
FA-27076

A proxy coalesces nearby requested spans into economical origin reads: caller slices use offsets relative to the containing fetch · case 01

A proxy coalesces nearby requested spans into economical origin reads: caller slices use offsets relative to the containing fetch.

HTTP ranges● Open access↗
FA-27077

A proxy coalesces nearby requested spans into economical origin reads: caller slices use offsets relative to the containing fetch · case 02

A proxy coalesces nearby requested spans into economical origin reads: caller slices use offsets relative to the containing fetch.

HTTP ranges◈ Members↗
FA-27078

A proxy coalesces nearby requested spans into economical origin reads: caller slices use offsets relative to the containing fetch · case 03

A proxy coalesces nearby requested spans into economical origin reads: caller slices use offsets relative to the containing fetch.

HTTP ranges◈ Members↗
FA-27079

A proxy coalesces nearby requested spans into economical origin reads: caller slices use offsets relative to the containing fetch · case 04

A proxy coalesces nearby requested spans into economical origin reads: caller slices use offsets relative to the containing fetch.

HTTP ranges◈ Members↗
FA-27080

A proxy coalesces nearby requested spans into economical origin reads: caller slices use offsets relative to the containing fetch · case 05

A proxy coalesces nearby requested spans into economical origin reads: caller slices use offsets relative to the containing fetch.

HTTP ranges◈ Members↗
FA-27081

A proxy coalesces nearby requested spans into economical origin reads: caller slice length excludes coalesced unrequested gap bytes · case 01

A proxy coalesces nearby requested spans into economical origin reads: caller slice length excludes coalesced unrequested gap bytes.

HTTP ranges● Open access↗
FA-27082

A proxy coalesces nearby requested spans into economical origin reads: caller slice length excludes coalesced unrequested gap bytes · case 02

A proxy coalesces nearby requested spans into economical origin reads: caller slice length excludes coalesced unrequested gap bytes.

HTTP ranges◈ Members↗
FA-27083

A proxy coalesces nearby requested spans into economical origin reads: caller slice length excludes coalesced unrequested gap bytes · case 03

A proxy coalesces nearby requested spans into economical origin reads: caller slice length excludes coalesced unrequested gap bytes.

HTTP ranges◈ Members↗
FA-27084

A proxy coalesces nearby requested spans into economical origin reads: caller slice length excludes coalesced unrequested gap bytes · case 04

A proxy coalesces nearby requested spans into economical origin reads: caller slice length excludes coalesced unrequested gap bytes.

HTTP ranges◈ Members↗
FA-27085

A proxy coalesces nearby requested spans into economical origin reads: caller slice length excludes coalesced unrequested gap bytes · case 05

A proxy coalesces nearby requested spans into economical origin reads: caller slice length excludes coalesced unrequested gap bytes.

HTTP ranges◈ Members↗
FA-27086

A proxy coalesces nearby requested spans into economical origin reads: empty selectors retain a null slot in caller mapping · case 01

A proxy coalesces nearby requested spans into economical origin reads: empty selectors retain a null slot in caller mapping.

HTTP ranges● Open access↗
FA-27087

A proxy coalesces nearby requested spans into economical origin reads: empty selectors retain a null slot in caller mapping · case 02

A proxy coalesces nearby requested spans into economical origin reads: empty selectors retain a null slot in caller mapping.

HTTP ranges◈ Members↗
FA-27088

A proxy coalesces nearby requested spans into economical origin reads: empty selectors retain a null slot in caller mapping · case 03

A proxy coalesces nearby requested spans into economical origin reads: empty selectors retain a null slot in caller mapping.

HTTP ranges◈ Members↗
FA-27089

A proxy coalesces nearby requested spans into economical origin reads: empty selectors retain a null slot in caller mapping · case 04

A proxy coalesces nearby requested spans into economical origin reads: empty selectors retain a null slot in caller mapping.

HTTP ranges◈ Members↗
FA-27090

A proxy coalesces nearby requested spans into economical origin reads: empty selectors retain a null slot in caller mapping · case 05

A proxy coalesces nearby requested spans into economical origin reads: empty selectors retain a null slot in caller mapping.

HTTP ranges◈ Members↗
FA-27091

A bounded Content-Range upload endpoint validates chunk admission: range uploads do not accept retrieval or unframed append methods · case 01

A bounded Content-Range upload endpoint validates chunk admission: range uploads do not accept retrieval or unframed append methods.

HTTP ranges● Open access↗
FA-27092

A bounded Content-Range upload endpoint validates chunk admission: range uploads do not accept retrieval or unframed append methods · case 02

A bounded Content-Range upload endpoint validates chunk admission: range uploads do not accept retrieval or unframed append methods.

HTTP ranges◈ Members↗
FA-27093

A bounded Content-Range upload endpoint validates chunk admission: range uploads do not accept retrieval or unframed append methods · case 03

A bounded Content-Range upload endpoint validates chunk admission: range uploads do not accept retrieval or unframed append methods.

HTTP ranges◈ Members↗
FA-27094

A bounded Content-Range upload endpoint validates chunk admission: range uploads do not accept retrieval or unframed append methods · case 04

A bounded Content-Range upload endpoint validates chunk admission: range uploads do not accept retrieval or unframed append methods.

HTTP ranges◈ Members↗
FA-27095

A bounded Content-Range upload endpoint validates chunk admission: range uploads do not accept retrieval or unframed append methods · case 05

A bounded Content-Range upload endpoint validates chunk admission: range uploads do not accept retrieval or unframed append methods.

HTTP ranges◈ Members↗
FA-27096

A bounded Content-Range upload endpoint validates chunk admission: a chunk cannot target another upload identifier · case 01

A bounded Content-Range upload endpoint validates chunk admission: a chunk cannot target another upload identifier.

HTTP ranges● Open access↗
FA-27097

A bounded Content-Range upload endpoint validates chunk admission: a chunk cannot target another upload identifier · case 02

A bounded Content-Range upload endpoint validates chunk admission: a chunk cannot target another upload identifier.

HTTP ranges◈ Members↗
FA-27098

A bounded Content-Range upload endpoint validates chunk admission: a chunk cannot target another upload identifier · case 03

A bounded Content-Range upload endpoint validates chunk admission: a chunk cannot target another upload identifier.

HTTP ranges◈ Members↗
FA-27099

A bounded Content-Range upload endpoint validates chunk admission: a chunk cannot target another upload identifier · case 04

A bounded Content-Range upload endpoint validates chunk admission: a chunk cannot target another upload identifier.

HTTP ranges◈ Members↗
FA-27100

A bounded Content-Range upload endpoint validates chunk admission: a chunk cannot target another upload identifier · case 05

A bounded Content-Range upload endpoint validates chunk admission: a chunk cannot target another upload identifier.

HTTP ranges◈ Members↗

INSPECTABLE BY DESIGN

Every result has a runnable source.

Runnable implementations with recorded outputs, source hashes, and explicit contracts. Related variants share a failure mechanism and belong together in evaluation splits.

Read the methodology ↗