Understand the failure.
Verify the repair.
Small, reproducible software failures. The broken implementation, the fix that didn’t work, and the one that passed—preserved together.
Explore the cases ↓How results are verified ↗WHAT THE ARCHIVE CONTAINS
100840 executable cases. 20168 are open.
Every case records the implementation that fails, the fix that did not work, and the repair that passed its checks—with recorded outputs and source hashes. This release adds 100840 cases across 20168 failure mechanisms and 254 domains.
The open tier gives you the failure and the unsuccessful fix for one case in every mechanism. The remaining 80672 cases, 5 variants per mechanism, are member-only: the verified repair, its recorded checks, and the full fixture suite are held in the member archive. Read the methodology ↗
A RECORD OF WHAT WENT WRONG
Browse the archive / 100840
Global holdout exclusion: Holdout percentage admits one extra bucket · case 01
A 5 percent holdout actually withholds 6 percent of users, and 0 percent still withholds bucket 0.
Global holdout exclusion: Holdout percentage admits one extra bucket · case 02
A 5 percent holdout actually withholds 6 percent of users, and 0 percent still withholds bucket 0.
Global holdout exclusion: Holdout percentage admits one extra bucket · case 03
A 5 percent holdout actually withholds 6 percent of users, and 0 percent still withholds bucket 0.
Global holdout exclusion: Holdout percentage admits one extra bucket · case 04
A 5 percent holdout actually withholds 6 percent of users, and 0 percent still withholds bucket 0.
Global holdout exclusion: Holdout percentage admits one extra bucket · case 05
A 5 percent holdout actually withholds 6 percent of users, and 0 percent still withholds bucket 0.
Semantic version clause: Numeric prerelease identifiers compare as text · case 01
beta.11 is treated as older than beta.2, so clients on the newer prerelease miss the feature.
Semantic version clause: Numeric prerelease identifiers compare as text · case 02
beta.11 is treated as older than beta.2, so clients on the newer prerelease miss the feature.
Semantic version clause: Numeric prerelease identifiers compare as text · case 03
beta.11 is treated as older than beta.2, so clients on the newer prerelease miss the feature.
Semantic version clause: Numeric prerelease identifiers compare as text · case 04
beta.11 is treated as older than beta.2, so clients on the newer prerelease miss the feature.
Semantic version clause: Numeric prerelease identifiers compare as text · case 05
beta.11 is treated as older than beta.2, so clients on the newer prerelease miss the feature.
Semantic version clause: Prereleases outrank their release · case 01
A "greater than 1.0.0-rc.1" rule excludes the final 1.0.0 release.
Semantic version clause: Prereleases outrank their release · case 02
A "greater than 1.0.0-rc.1" rule excludes the final 1.0.0 release.
Semantic version clause: Prereleases outrank their release · case 03
A "greater than 1.0.0-rc.1" rule excludes the final 1.0.0 release.
Semantic version clause: Prereleases outrank their release · case 04
A "greater than 1.0.0-rc.1" rule excludes the final 1.0.0 release.
Semantic version clause: Prereleases outrank their release · case 05
A "greater than 1.0.0-rc.1" rule excludes the final 1.0.0 release.
Semantic version clause: Short versions are rejected · case 01
Clients reporting "2.1" never match any version rule.
Semantic version clause: Short versions are rejected · case 02
Clients reporting "2.1" never match any version rule.
Semantic version clause: Short versions are rejected · case 03
Clients reporting "2.1" never match any version rule.
Semantic version clause: Short versions are rejected · case 04
Clients reporting "2.1" never match any version rule.
Semantic version clause: Short versions are rejected · case 05
Clients reporting "2.1" never match any version rule.
Semantic version clause: Build metadata makes versions invalid · case 01
Versions such as 1.0.0+build.5 fail every version clause.
Semantic version clause: Build metadata makes versions invalid · case 02
Versions such as 1.0.0+build.5 fail every version clause.
Semantic version clause: Build metadata makes versions invalid · case 03
Versions such as 1.0.0+build.5 fail every version clause.
Semantic version clause: Build metadata makes versions invalid · case 04
Versions such as 1.0.0+build.5 fail every version clause.
Semantic version clause: Build metadata makes versions invalid · case 05
Versions such as 1.0.0+build.5 fail every version clause.
Semantic version clause: Prerelease prefixes compare equal · case 01
1.0.0-alpha and 1.0.0-alpha.1 are treated as the same version.
Semantic version clause: Prerelease prefixes compare equal · case 02
1.0.0-alpha and 1.0.0-alpha.1 are treated as the same version.
Semantic version clause: Prerelease prefixes compare equal · case 03
1.0.0-alpha and 1.0.0-alpha.1 are treated as the same version.
Semantic version clause: Prerelease prefixes compare equal · case 04
1.0.0-alpha and 1.0.0-alpha.1 are treated as the same version.
Semantic version clause: Prerelease prefixes compare equal · case 05
1.0.0-alpha and 1.0.0-alpha.1 are treated as the same version.
Multi-context canonical key: Colons are escaped before percent signs · case 01
Keys containing ":" become "%253A", so multi-context keys disagree with other SDKs.
Multi-context canonical key: Colons are escaped before percent signs · case 02
Keys containing ":" become "%253A", so multi-context keys disagree with other SDKs.
Multi-context canonical key: Colons are escaped before percent signs · case 03
Keys containing ":" become "%253A", so multi-context keys disagree with other SDKs.
Multi-context canonical key: Colons are escaped before percent signs · case 04
Keys containing ":" become "%253A", so multi-context keys disagree with other SDKs.
Multi-context canonical key: Colons are escaped before percent signs · case 05
Keys containing ":" become "%253A", so multi-context keys disagree with other SDKs.
Multi-context canonical key: Context kinds follow insertion order · case 01
The same user+org context built in a different order buckets differently.
Multi-context canonical key: Context kinds follow insertion order · case 02
The same user+org context built in a different order buckets differently.
Multi-context canonical key: Context kinds follow insertion order · case 03
The same user+org context built in a different order buckets differently.
Multi-context canonical key: Context kinds follow insertion order · case 04
The same user+org context built in a different order buckets differently.
Multi-context canonical key: Context kinds follow insertion order · case 05
The same user+org context built in a different order buckets differently.
Multi-context canonical key: Any single-kind context uses a bare key · case 01
An org-only context collides with a user whose key equals the org key.
Multi-context canonical key: Any single-kind context uses a bare key · case 02
An org-only context collides with a user whose key equals the org key.
Multi-context canonical key: Any single-kind context uses a bare key · case 03
An org-only context collides with a user whose key equals the org key.
Multi-context canonical key: Any single-kind context uses a bare key · case 04
An org-only context collides with a user whose key equals the org key.
Multi-context canonical key: Any single-kind context uses a bare key · case 05
An org-only context collides with a user whose key equals the org key.
Multi-context canonical key: Empty context keys are accepted · case 01
Contexts with a blank org key all share the canonical key "org:".
Multi-context canonical key: Empty context keys are accepted · case 02
Contexts with a blank org key all share the canonical key "org:".
Multi-context canonical key: Empty context keys are accepted · case 03
Contexts with a blank org key all share the canonical key "org:".
Multi-context canonical key: Empty context keys are accepted · case 04
Contexts with a blank org key all share the canonical key "org:".
Multi-context canonical key: Empty context keys are accepted · case 05
Contexts with a blank org key all share the canonical key "org:".
Multi-context canonical key: Bare user keys are escaped · case 01
Single-user contexts with ":" in the key no longer match the legacy user key.
Multi-context canonical key: Bare user keys are escaped · case 02
Single-user contexts with ":" in the key no longer match the legacy user key.
Multi-context canonical key: Bare user keys are escaped · case 03
Single-user contexts with ":" in the key no longer match the legacy user key.
Multi-context canonical key: Bare user keys are escaped · case 04
Single-user contexts with ":" in the key no longer match the legacy user key.
Multi-context canonical key: Bare user keys are escaped · case 05
Single-user contexts with ":" in the key no longer match the legacy user key.
Network allowlist clause: The prefix mask selects low-order bits · case 01
A /24 rule compares the last three octets, so unrelated networks match.
Network allowlist clause: The prefix mask selects low-order bits · case 02
A /24 rule compares the last three octets, so unrelated networks match.
Network allowlist clause: The prefix mask selects low-order bits · case 03
A /24 rule compares the last three octets, so unrelated networks match.
Network allowlist clause: The prefix mask selects low-order bits · case 04
A /24 rule compares the last three octets, so unrelated networks match.
Network allowlist clause: The prefix mask selects low-order bits · case 05
A /24 rule compares the last three octets, so unrelated networks match.
Network allowlist clause: Octets above 255 are accepted · case 01
An entry such as 1.0.0.256/8 is parsed into an unrelated network instead of being skipped.
Network allowlist clause: Octets above 255 are accepted · case 02
An entry such as 1.0.0.256/8 is parsed into an unrelated network instead of being skipped.
Network allowlist clause: Octets above 255 are accepted · case 03
An entry such as 1.0.0.256/8 is parsed into an unrelated network instead of being skipped.
Network allowlist clause: Octets above 255 are accepted · case 04
An entry such as 1.0.0.256/8 is parsed into an unrelated network instead of being skipped.
Network allowlist clause: Octets above 255 are accepted · case 05
An entry such as 1.0.0.256/8 is parsed into an unrelated network instead of being skipped.
Network allowlist clause: A bare address allows every source · case 01
Listing a single office IP without "/32" opens the feature to the whole internet.
Network allowlist clause: A bare address allows every source · case 02
Listing a single office IP without "/32" opens the feature to the whole internet.
Network allowlist clause: A bare address allows every source · case 03
Listing a single office IP without "/32" opens the feature to the whole internet.
Network allowlist clause: A bare address allows every source · case 04
Listing a single office IP without "/32" opens the feature to the whole internet.
Network allowlist clause: A bare address allows every source · case 05
Listing a single office IP without "/32" opens the feature to the whole internet.
Network allowlist clause: Entries with host bits never match · case 01
An entry written as 10.1.2.77/24 silently matches nothing.
Network allowlist clause: Entries with host bits never match · case 02
An entry written as 10.1.2.77/24 silently matches nothing.
Network allowlist clause: Entries with host bits never match · case 03
An entry written as 10.1.2.77/24 silently matches nothing.
Network allowlist clause: Entries with host bits never match · case 04
An entry written as 10.1.2.77/24 silently matches nothing.
Network allowlist clause: Entries with host bits never match · case 05
An entry written as 10.1.2.77/24 silently matches nothing.
Exposure event deduplication: Variant is missing from the dedupe key · case 01
A user re-bucketed to another variant within the window has no exposure logged for the new variant.
Exposure event deduplication: Variant is missing from the dedupe key · case 02
A user re-bucketed to another variant within the window has no exposure logged for the new variant.
Exposure event deduplication: Variant is missing from the dedupe key · case 03
A user re-bucketed to another variant within the window has no exposure logged for the new variant.
Exposure event deduplication: Variant is missing from the dedupe key · case 04
A user re-bucketed to another variant within the window has no exposure logged for the new variant.
Exposure event deduplication: Variant is missing from the dedupe key · case 05
A user re-bucketed to another variant within the window has no exposure logged for the new variant.
Exposure event deduplication: Suppressed exposures slide the window · case 01
A frequently active user is exposed once and then never again.
Exposure event deduplication: Suppressed exposures slide the window · case 02
A frequently active user is exposed once and then never again.
Exposure event deduplication: Suppressed exposures slide the window · case 03
A frequently active user is exposed once and then never again.
Exposure event deduplication: Suppressed exposures slide the window · case 04
A frequently active user is exposed once and then never again.
Exposure event deduplication: Suppressed exposures slide the window · case 05
A frequently active user is exposed once and then never again.
Exposure event deduplication: An exposure exactly one window later is suppressed · case 01
Hourly exposure refresh drifts because the event at exactly one window is dropped.
Exposure event deduplication: An exposure exactly one window later is suppressed · case 02
Hourly exposure refresh drifts because the event at exactly one window is dropped.
Exposure event deduplication: An exposure exactly one window later is suppressed · case 03
Hourly exposure refresh drifts because the event at exactly one window is dropped.
Exposure event deduplication: An exposure exactly one window later is suppressed · case 04
Hourly exposure refresh drifts because the event at exactly one window is dropped.
Exposure event deduplication: An exposure exactly one window later is suppressed · case 05
Hourly exposure refresh drifts because the event at exactly one window is dropped.
Exposure event deduplication: Events are processed in arrival order · case 01
Late-arriving older events are emitted after newer ones and suppress the wrong exposures.
Exposure event deduplication: Events are processed in arrival order · case 02
Late-arriving older events are emitted after newer ones and suppress the wrong exposures.
Exposure event deduplication: Events are processed in arrival order · case 03
Late-arriving older events are emitted after newer ones and suppress the wrong exposures.
Exposure event deduplication: Events are processed in arrival order · case 04
Late-arriving older events are emitted after newer ones and suppress the wrong exposures.
Exposure event deduplication: Events are processed in arrival order · case 05
Late-arriving older events are emitted after newer ones and suppress the wrong exposures.
Guarded progressive ramp: Low-sample intervals trigger rollbacks · case 01
A single early error among a handful of requests rolls back a healthy launch.
Guarded progressive ramp: Low-sample intervals trigger rollbacks · case 02
A single early error among a handful of requests rolls back a healthy launch.
Guarded progressive ramp: Low-sample intervals trigger rollbacks · case 03
A single early error among a handful of requests rolls back a healthy launch.
Guarded progressive ramp: Low-sample intervals trigger rollbacks · case 04
A single early error among a handful of requests rolls back a healthy launch.
Guarded progressive ramp: Low-sample intervals trigger rollbacks · case 05
A single early error among a handful of requests rolls back a healthy launch.
INSPECTABLE BY DESIGN
Every result has a runnable source.
Runnable implementations with recorded outputs, source hashes, and explicit contracts. Related variants share a failure mechanism and belong together in evaluation splits.
Read the methodology ↗