FA-1581 / Reliability / Open access

Cut traffic to a replacement region: Both regions accept writes during a single-writer cutover · case 01

The regional cutover operation is admitted even though both regions accept writes during a single-writer cutover.

Verified by executionVariant 1 · 9 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The admission path omits the source writes stopped invariant while validating the other operation preconditions.

THE FAILURE

The admission path omits the source writes stopped invariant while validating the other operation preconditions.

Unsuccessful approach: Adding the source writes stopped check repairs the reported defect, but replacing the adjacent dependency ready check loses that independent invariant.

Case contract

Return a Boolean admission decision for cut traffic to a replacement region. The record r must satisfy all of: r['replication_caught_up'][0] >= r['replication_caught_up'][1]; r['source_writes_stopped'] is True; all(r['dependency_ready']); r['capacity_headroom'][0]+r['capacity_headroom'][1] <= r['capacity_headroom'][2]; r['cutover_generation'][0] > r['cutover_generation'][1]. Extra tracing fields are ignored; validation does not mutate the record.

Why this case matters

A deterministic local contract for reliability. Each negative fixture violates exactly one invariant. No transport timing, persistence, cryptographic verification, or full protocol implementation is claimed.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(r):
    return (r['replication_caught_up'][0] >= r['replication_caught_up'][1]) and (all(r['dependency_ready'])) and (r['capacity_headroom'][0]+r['capacity_headroom'][1] <= r['capacity_headroom'][2]) and (r['cutover_generation'][0] > r['cutover_generation'][1])
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
r = {'replication_caught_up': [30, 30], 'source_writes_stopped': True, 'dependency_ready': [True, True], 'capacity_headroom': [4, 5, 10], 'cutover_generation': [8, 7]}
check('valid operation', solve(r), True)
check('Cutover loses writes beyond the destination applied watermark', solve(dict(r, **{'replication_caught_up': [29, 30]})), False)
check('Both regions accept writes during a single-writer cutover', solve(dict(r, **{'source_writes_stopped': False})), False)
check('Application traffic arrives before regional dependencies are ready', solve(dict(r, **{'dependency_ready': [True, False]})), False)
check('Destination capacity cannot serve the transferred load', solve(dict(r, **{'capacity_headroom': [6, 5, 10]})), False)
check('A delayed cutover command reverses a newer regional decision', solve(dict(r, **{'cutover_generation': [7, 7]})), False)
check('unrelated tracing metadata', solve(dict(r, trace='run-'+str(N))), True)
check('repeat validation is pure', solve(r), True)
invalid = {'replication_caught_up': [29, 30], 'source_writes_stopped': False, 'dependency_ready': [True, False], 'capacity_headroom': [6, 5, 10], 'cutover_generation': [7, 7]}
keys = list(invalid)
pair = {keys[N % len(keys)]: invalid[keys[N % len(keys)]], keys[(N+1) % len(keys)]: invalid[keys[(N+1) % len(keys)]]}
check('two independent violations in variant', solve(dict(r, **pair)), False)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
valid operation	`True`	`True`	Passed
Cutover loses writes beyond the destination applied watermark	`False`	`False`	Passed
Both regions accept writes during a single-writer cutover	`True`	`False`	Failed
Application traffic arrives before regional dependencies are ready	`False`	`False`	Passed
Destination capacity cannot serve the transferred load	`False`	`False`	Passed
A delayed cutover command reverses a newer regional decision	`False`	`False`	Passed
unrelated tracing metadata	`True`	`True`	Passed
repeat validation is pure	`True`	`True`	Passed
two independent violations in variant	`False`	`False`	Passed

SHA-256 / d3198e9576748fe4c0f6a77524127afb9ecffe1a8f0646af1c30f93de0d9b47a

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(r):
    return (r['replication_caught_up'][0] >= r['replication_caught_up'][1]) and (r['source_writes_stopped'] is True) and (r['capacity_headroom'][0]+r['capacity_headroom'][1] <= r['capacity_headroom'][2]) and (r['cutover_generation'][0] > r['cutover_generation'][1])
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
r = {'replication_caught_up': [30, 30], 'source_writes_stopped': True, 'dependency_ready': [True, True], 'capacity_headroom': [4, 5, 10], 'cutover_generation': [8, 7]}
check('valid operation', solve(r), True)
check('Cutover loses writes beyond the destination applied watermark', solve(dict(r, **{'replication_caught_up': [29, 30]})), False)
check('Both regions accept writes during a single-writer cutover', solve(dict(r, **{'source_writes_stopped': False})), False)
check('Application traffic arrives before regional dependencies are ready', solve(dict(r, **{'dependency_ready': [True, False]})), False)
check('Destination capacity cannot serve the transferred load', solve(dict(r, **{'capacity_headroom': [6, 5, 10]})), False)
check('A delayed cutover command reverses a newer regional decision', solve(dict(r, **{'cutover_generation': [7, 7]})), False)
check('unrelated tracing metadata', solve(dict(r, trace='run-'+str(N))), True)
check('repeat validation is pure', solve(r), True)
invalid = {'replication_caught_up': [29, 30], 'source_writes_stopped': False, 'dependency_ready': [True, False], 'capacity_headroom': [6, 5, 10], 'cutover_generation': [7, 7]}
keys = list(invalid)
pair = {keys[N % len(keys)]: invalid[keys[N % len(keys)]], keys[(N+1) % len(keys)]: invalid[keys[(N+1) % len(keys)]]}
check('two independent violations in variant', solve(dict(r, **pair)), False)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
valid operation	`True`	`True`	Passed
Cutover loses writes beyond the destination applied watermark	`False`	`False`	Passed
Both regions accept writes during a single-writer cutover	`False`	`False`	Passed
Application traffic arrives before regional dependencies are ready	`True`	`False`	Failed
Destination capacity cannot serve the transferred load	`False`	`False`	Passed
A delayed cutover command reverses a newer regional decision	`False`	`False`	Passed
unrelated tracing metadata	`True`	`True`	Passed
repeat validation is pure	`True`	`True`	Passed
two independent violations in variant	`False`	`False`	Passed

SHA-256 / 397a89a05a3da2a283bdb3a9d8a146a7811df567e2bb1aa63e0d9060b8980b01

HELD IN THE MEMBER ARCHIVE

The verified repair and its recorded checks are member-only.

This mechanism has 9 recorded checks per implementation. The open-access tier publishes the failure and the unsuccessful fix; the repaired source that passes every check, and the observations that prove it, are available to members.

Every case sharing this mechanism uses the same contract and the same repair, so this one record is held back for all of them.

Member access is invitation-based. Sign in with your invited account to inspect the repair.

Verification & scope

This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:37:03.780740+00:00.

Case digest / 7d1f679902c177a75568afeaadcdb88e8ab4bafd238f6b795246c23b6991cef8