Understand the failure.
Verify the repair.
Small, reproducible software failures. The broken implementation, the fix that didn’t work, and the one that passed—preserved together.
Explore the cases ↓How results are verified ↗WHAT THE ARCHIVE CONTAINS
100840 executable cases. 20168 are open.
Every case records the implementation that fails, the fix that did not work, and the repair that passed its checks—with recorded outputs and source hashes. This release adds 100840 cases across 20168 failure mechanisms and 254 domains.
The open tier gives you the failure and the unsuccessful fix for one case in every mechanism. The remaining 80672 cases, 5 variants per mechanism, are member-only: the verified repair, its recorded checks, and the full fixture suite are held in the member archive. Read the methodology ↗
A RECORD OF WHAT WENT WRONG
Browse the archive / 100840
Scheduled rollout ramp: Hold steps reset the rollout to zero · case 01
A schedule pause entry turns the feature off instead of holding the current percentage.
Scheduled rollout ramp: Hold steps reset the rollout to zero · case 02
A schedule pause entry turns the feature off instead of holding the current percentage.
Scheduled rollout ramp: Hold steps reset the rollout to zero · case 03
A schedule pause entry turns the feature off instead of holding the current percentage.
Scheduled rollout ramp: Hold steps reset the rollout to zero · case 04
A schedule pause entry turns the feature off instead of holding the current percentage.
Scheduled rollout ramp: Hold steps reset the rollout to zero · case 05
A schedule pause entry turns the feature off instead of holding the current percentage.
Scheduled rollout ramp: Negative step percentages pass through · case 01
A mistyped -20 percent step reports a negative rollout that downstream gates treat inconsistently.
Scheduled rollout ramp: Negative step percentages pass through · case 02
A mistyped -20 percent step reports a negative rollout that downstream gates treat inconsistently.
Scheduled rollout ramp: Negative step percentages pass through · case 03
A mistyped -20 percent step reports a negative rollout that downstream gates treat inconsistently.
Scheduled rollout ramp: Negative step percentages pass through · case 04
A mistyped -20 percent step reports a negative rollout that downstream gates treat inconsistently.
Scheduled rollout ramp: Negative step percentages pass through · case 05
A mistyped -20 percent step reports a negative rollout that downstream gates treat inconsistently.
Scheduled rollout ramp: Active step is reported by sorted position · case 01
Dashboards highlight the wrong schedule row as the active step.
Scheduled rollout ramp: Active step is reported by sorted position · case 02
Dashboards highlight the wrong schedule row as the active step.
Scheduled rollout ramp: Active step is reported by sorted position · case 03
Dashboards highlight the wrong schedule row as the active step.
Scheduled rollout ramp: Active step is reported by sorted position · case 04
Dashboards highlight the wrong schedule row as the active step.
Scheduled rollout ramp: Active step is reported by sorted position · case 05
Dashboards highlight the wrong schedule row as the active step.
Flag value cache freshness: Cached values survive their full ttl second · case 01
A value exactly 30 seconds old is still served from cache.
Flag value cache freshness: Cached values survive their full ttl second · case 02
A value exactly 30 seconds old is still served from cache.
Flag value cache freshness: Cached values survive their full ttl second · case 03
A value exactly 30 seconds old is still served from cache.
Flag value cache freshness: Cached values survive their full ttl second · case 04
A value exactly 30 seconds old is still served from cache.
Flag value cache freshness: Cached values survive their full ttl second · case 05
A value exactly 30 seconds old is still served from cache.
Flag value cache freshness: Refetch merges into the old cache · case 01
A flag deleted on the server keeps serving its last cached value.
Flag value cache freshness: Refetch merges into the old cache · case 02
A flag deleted on the server keeps serving its last cached value.
Flag value cache freshness: Refetch merges into the old cache · case 03
A flag deleted on the server keeps serving its last cached value.
Flag value cache freshness: Refetch merges into the old cache · case 04
A flag deleted on the server keeps serving its last cached value.
Flag value cache freshness: Refetch merges into the old cache · case 05
A flag deleted on the server keeps serving its last cached value.
Flag value cache freshness: Kill switches survive a refetch · case 01
After an operator fixes a flag and clients refetch, the old kill switch keeps forcing defaults.
Flag value cache freshness: Kill switches survive a refetch · case 02
After an operator fixes a flag and clients refetch, the old kill switch keeps forcing defaults.
Flag value cache freshness: Kill switches survive a refetch · case 03
After an operator fixes a flag and clients refetch, the old kill switch keeps forcing defaults.
Flag value cache freshness: Kill switches survive a refetch · case 04
After an operator fixes a flag and clients refetch, the old kill switch keeps forcing defaults.
Flag value cache freshness: Kill switches survive a refetch · case 05
After an operator fixes a flag and clients refetch, the old kill switch keeps forcing defaults.
Flag value cache freshness: Staleness masks an active kill switch · case 01
A killed flag is reported as stale, hiding the operator action in diagnostics.
Flag value cache freshness: Staleness masks an active kill switch · case 02
A killed flag is reported as stale, hiding the operator action in diagnostics.
Flag value cache freshness: Staleness masks an active kill switch · case 03
A killed flag is reported as stale, hiding the operator action in diagnostics.
Flag value cache freshness: Staleness masks an active kill switch · case 04
A killed flag is reported as stale, hiding the operator action in diagnostics.
Flag value cache freshness: Staleness masks an active kill switch · case 05
A killed flag is reported as stale, hiding the operator action in diagnostics.
Flag value cache freshness: Only the first fetch starts the ttl clock · case 01
Clients go stale 30 seconds after startup even though they refetch continuously.
Flag value cache freshness: Only the first fetch starts the ttl clock · case 02
Clients go stale 30 seconds after startup even though they refetch continuously.
Flag value cache freshness: Only the first fetch starts the ttl clock · case 03
Clients go stale 30 seconds after startup even though they refetch continuously.
Flag value cache freshness: Only the first fetch starts the ttl clock · case 04
Clients go stale 30 seconds after startup even though they refetch continuously.
Flag value cache freshness: Only the first fetch starts the ttl clock · case 05
Clients go stale 30 seconds after startup even though they refetch continuously.
Sticky variant persistence: Records from older experiment versions stay sticky · case 01
After an experiment restart users keep the arm from the previous version.
Sticky variant persistence: Records from older experiment versions stay sticky · case 02
After an experiment restart users keep the arm from the previous version.
Sticky variant persistence: Records from older experiment versions stay sticky · case 03
After an experiment restart users keep the arm from the previous version.
Sticky variant persistence: Records from older experiment versions stay sticky · case 04
After an experiment restart users keep the arm from the previous version.
Sticky variant persistence: Records from older experiment versions stay sticky · case 05
After an experiment restart users keep the arm from the previous version.
Sticky variant persistence: Retired variants remain sticky · case 01
Users keep receiving a variant that has been removed from the experiment.
Sticky variant persistence: Retired variants remain sticky · case 02
Users keep receiving a variant that has been removed from the experiment.
Sticky variant persistence: Retired variants remain sticky · case 03
Users keep receiving a variant that has been removed from the experiment.
Sticky variant persistence: Retired variants remain sticky · case 04
Users keep receiving a variant that has been removed from the experiment.
Sticky variant persistence: Retired variants remain sticky · case 05
Users keep receiving a variant that has been removed from the experiment.
Sticky variant persistence: Traffic exclusion erases the stored assignment · case 01
Temporarily shrinking traffic loses users' history, so they are reshuffled when traffic returns.
Sticky variant persistence: Traffic exclusion erases the stored assignment · case 02
Temporarily shrinking traffic loses users' history, so they are reshuffled when traffic returns.
Sticky variant persistence: Traffic exclusion erases the stored assignment · case 03
Temporarily shrinking traffic loses users' history, so they are reshuffled when traffic returns.
Sticky variant persistence: Traffic exclusion erases the stored assignment · case 04
Temporarily shrinking traffic loses users' history, so they are reshuffled when traffic returns.
Sticky variant persistence: Traffic exclusion erases the stored assignment · case 05
Temporarily shrinking traffic loses users' history, so they are reshuffled when traffic returns.
Sticky variant persistence: Newer stored versions are accepted · case 01
After rolling an experiment back, records written by the abandoned newer version stay sticky.
Sticky variant persistence: Newer stored versions are accepted · case 02
After rolling an experiment back, records written by the abandoned newer version stay sticky.
Sticky variant persistence: Newer stored versions are accepted · case 03
After rolling an experiment back, records written by the abandoned newer version stay sticky.
Sticky variant persistence: Newer stored versions are accepted · case 04
After rolling an experiment back, records written by the abandoned newer version stay sticky.
Sticky variant persistence: Newer stored versions are accepted · case 05
After rolling an experiment back, records written by the abandoned newer version stay sticky.
Sticky variant persistence: Exclusion is checked before sticky records · case 01
Reducing traffic allocation drops users who already had a valid sticky assignment.
Sticky variant persistence: Exclusion is checked before sticky records · case 02
Reducing traffic allocation drops users who already had a valid sticky assignment.
Sticky variant persistence: Exclusion is checked before sticky records · case 03
Reducing traffic allocation drops users who already had a valid sticky assignment.
Sticky variant persistence: Exclusion is checked before sticky records · case 04
Reducing traffic allocation drops users who already had a valid sticky assignment.
Sticky variant persistence: Exclusion is checked before sticky records · case 05
Reducing traffic allocation drops users who already had a valid sticky assignment.
Mutually exclusive layer allocation: Range ends are inclusive · case 01
The user at a shared boundary is claimed by the earlier experiment, so adjacent ranges overlap.
Mutually exclusive layer allocation: Range ends are inclusive · case 02
The user at a shared boundary is claimed by the earlier experiment, so adjacent ranges overlap.
Mutually exclusive layer allocation: Range ends are inclusive · case 03
The user at a shared boundary is claimed by the earlier experiment, so adjacent ranges overlap.
Mutually exclusive layer allocation: Range ends are inclusive · case 04
The user at a shared boundary is claimed by the earlier experiment, so adjacent ranges overlap.
Mutually exclusive layer allocation: Range ends are inclusive · case 05
The user at a shared boundary is claimed by the earlier experiment, so adjacent ranges overlap.
Mutually exclusive layer allocation: Paused experiments release their range · case 01
Pausing an experiment pours its users into an overlapping experiment listed later.
Mutually exclusive layer allocation: Paused experiments release their range · case 02
Pausing an experiment pours its users into an overlapping experiment listed later.
Mutually exclusive layer allocation: Paused experiments release their range · case 03
Pausing an experiment pours its users into an overlapping experiment listed later.
Mutually exclusive layer allocation: Paused experiments release their range · case 04
Pausing an experiment pours its users into an overlapping experiment listed later.
Mutually exclusive layer allocation: Paused experiments release their range · case 05
Pausing an experiment pours its users into an overlapping experiment listed later.
Mutually exclusive layer allocation: The last overlapping range wins · case 01
Adding a new experiment at the end of a layer silently steals users from earlier ones.
Mutually exclusive layer allocation: The last overlapping range wins · case 02
Adding a new experiment at the end of a layer silently steals users from earlier ones.
Mutually exclusive layer allocation: The last overlapping range wins · case 03
Adding a new experiment at the end of a layer silently steals users from earlier ones.
Mutually exclusive layer allocation: The last overlapping range wins · case 04
Adding a new experiment at the end of a layer silently steals users from earlier ones.
Mutually exclusive layer allocation: The last overlapping range wins · case 05
Adding a new experiment at the end of a layer silently steals users from earlier ones.
Mutually exclusive layer allocation: Percent bounds are scaled to basis points · case 01
Every range reaches ten times too far, so the first experiment captures nearly the whole layer.
Mutually exclusive layer allocation: Percent bounds are scaled to basis points · case 02
Every range reaches ten times too far, so the first experiment captures nearly the whole layer.
Mutually exclusive layer allocation: Percent bounds are scaled to basis points · case 03
Every range reaches ten times too far, so the first experiment captures nearly the whole layer.
Mutually exclusive layer allocation: Percent bounds are scaled to basis points · case 04
Every range reaches ten times too far, so the first experiment captures nearly the whole layer.
Mutually exclusive layer allocation: Percent bounds are scaled to basis points · case 05
Every range reaches ten times too far, so the first experiment captures nearly the whole layer.
Mutually exclusive layer allocation: Layer hash drops the separator · case 01
Layer salts and user ids run together, so ("ab", "c") and ("a", "bc") share a bucket and SDKs disagree.
Mutually exclusive layer allocation: Layer hash drops the separator · case 02
Layer salts and user ids run together, so ("ab", "c") and ("a", "bc") share a bucket and SDKs disagree.
Mutually exclusive layer allocation: Layer hash drops the separator · case 03
Layer salts and user ids run together, so ("ab", "c") and ("a", "bc") share a bucket and SDKs disagree.
Mutually exclusive layer allocation: Layer hash drops the separator · case 04
Layer salts and user ids run together, so ("ab", "c") and ("a", "bc") share a bucket and SDKs disagree.
Mutually exclusive layer allocation: Layer hash drops the separator · case 05
Layer salts and user ids run together, so ("ab", "c") and ("a", "bc") share a bucket and SDKs disagree.
Global holdout exclusion: Holdout overrides explicit forcing · case 01
QA accounts that happen to hash into the holdout cannot be forced into treatment.
Global holdout exclusion: Holdout overrides explicit forcing · case 02
QA accounts that happen to hash into the holdout cannot be forced into treatment.
Global holdout exclusion: Holdout overrides explicit forcing · case 03
QA accounts that happen to hash into the holdout cannot be forced into treatment.
Global holdout exclusion: Holdout overrides explicit forcing · case 04
QA accounts that happen to hash into the holdout cannot be forced into treatment.
Global holdout exclusion: Holdout overrides explicit forcing · case 05
QA accounts that happen to hash into the holdout cannot be forced into treatment.
Global holdout exclusion: The holdout reuses the flag bucket · case 01
Holdout membership differs per flag, so every user is held out of some launches and exposed to others.
Global holdout exclusion: The holdout reuses the flag bucket · case 02
Holdout membership differs per flag, so every user is held out of some launches and exposed to others.
Global holdout exclusion: The holdout reuses the flag bucket · case 03
Holdout membership differs per flag, so every user is held out of some launches and exposed to others.
Global holdout exclusion: The holdout reuses the flag bucket · case 04
Holdout membership differs per flag, so every user is held out of some launches and exposed to others.
Global holdout exclusion: The holdout reuses the flag bucket · case 05
Holdout membership differs per flag, so every user is held out of some launches and exposed to others.
INSPECTABLE BY DESIGN
Every result has a runnable source.
Runnable implementations with recorded outputs, source hashes, and explicit contracts. Related variants share a failure mechanism and belong together in evaluation splits.
Read the methodology ↗