FA-1591 / Reliability / Open access

Cut traffic to a replacement region: Destination capacity cannot serve the transferred load · case 01

The regional cutover operation is admitted even though destination capacity cannot serve the transferred load.

Verified by executionVariant 1 · 9 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The admission path omits the capacity headroom invariant while validating the other operation preconditions.

VERIFIED REPAIR

Require r['capacity_headroom'][0]+r['capacity_headroom'][1] <= r['capacity_headroom'][2] together with every other stated precondition before accepting the operation.

Unsuccessful approach: Adding the capacity headroom check repairs the reported defect, but replacing the adjacent cutover generation check loses that independent invariant.

Case contract

Return a Boolean admission decision for cut traffic to a replacement region. The record r must satisfy all of: r['replication_caught_up'][0] >= r['replication_caught_up'][1]; r['source_writes_stopped'] is True; all(r['dependency_ready']); r['capacity_headroom'][0]+r['capacity_headroom'][1] <= r['capacity_headroom'][2]; r['cutover_generation'][0] > r['cutover_generation'][1]. Extra tracing fields are ignored; validation does not mutate the record.

Why this case matters

A deterministic local contract for reliability. Each negative fixture violates exactly one invariant. No transport timing, persistence, cryptographic verification, or full protocol implementation is claimed.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(r):
    return (r['replication_caught_up'][0] >= r['replication_caught_up'][1]) and (r['source_writes_stopped'] is True) and (all(r['dependency_ready'])) and (r['cutover_generation'][0] > r['cutover_generation'][1])
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
r = {'replication_caught_up': [30, 30], 'source_writes_stopped': True, 'dependency_ready': [True, True], 'capacity_headroom': [4, 5, 10], 'cutover_generation': [8, 7]}
check('valid operation', solve(r), True)
check('Cutover loses writes beyond the destination applied watermark', solve(dict(r, **{'replication_caught_up': [29, 30]})), False)
check('Both regions accept writes during a single-writer cutover', solve(dict(r, **{'source_writes_stopped': False})), False)
check('Application traffic arrives before regional dependencies are ready', solve(dict(r, **{'dependency_ready': [True, False]})), False)
check('Destination capacity cannot serve the transferred load', solve(dict(r, **{'capacity_headroom': [6, 5, 10]})), False)
check('A delayed cutover command reverses a newer regional decision', solve(dict(r, **{'cutover_generation': [7, 7]})), False)
check('unrelated tracing metadata', solve(dict(r, trace='run-'+str(N))), True)
check('repeat validation is pure', solve(r), True)
invalid = {'replication_caught_up': [29, 30], 'source_writes_stopped': False, 'dependency_ready': [True, False], 'capacity_headroom': [6, 5, 10], 'cutover_generation': [7, 7]}
keys = list(invalid)
pair = {keys[N % len(keys)]: invalid[keys[N % len(keys)]], keys[(N+1) % len(keys)]: invalid[keys[(N+1) % len(keys)]]}
check('two independent violations in variant', solve(dict(r, **pair)), False)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
valid operation	`True`	`True`	Passed
Cutover loses writes beyond the destination applied watermark	`False`	`False`	Passed
Both regions accept writes during a single-writer cutover	`False`	`False`	Passed
Application traffic arrives before regional dependencies are ready	`False`	`False`	Passed
Destination capacity cannot serve the transferred load	`True`	`False`	Failed
A delayed cutover command reverses a newer regional decision	`False`	`False`	Passed
unrelated tracing metadata	`True`	`True`	Passed
repeat validation is pure	`True`	`True`	Passed
two independent violations in variant	`False`	`False`	Passed

SHA-256 / d5d3112b9ac929f3db0b21a0183ea304edbba58886bcc65236f1f70de7180a3c

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(r):
    return (r['replication_caught_up'][0] >= r['replication_caught_up'][1]) and (r['source_writes_stopped'] is True) and (all(r['dependency_ready'])) and (r['capacity_headroom'][0]+r['capacity_headroom'][1] <= r['capacity_headroom'][2])
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
r = {'replication_caught_up': [30, 30], 'source_writes_stopped': True, 'dependency_ready': [True, True], 'capacity_headroom': [4, 5, 10], 'cutover_generation': [8, 7]}
check('valid operation', solve(r), True)
check('Cutover loses writes beyond the destination applied watermark', solve(dict(r, **{'replication_caught_up': [29, 30]})), False)
check('Both regions accept writes during a single-writer cutover', solve(dict(r, **{'source_writes_stopped': False})), False)
check('Application traffic arrives before regional dependencies are ready', solve(dict(r, **{'dependency_ready': [True, False]})), False)
check('Destination capacity cannot serve the transferred load', solve(dict(r, **{'capacity_headroom': [6, 5, 10]})), False)
check('A delayed cutover command reverses a newer regional decision', solve(dict(r, **{'cutover_generation': [7, 7]})), False)
check('unrelated tracing metadata', solve(dict(r, trace='run-'+str(N))), True)
check('repeat validation is pure', solve(r), True)
invalid = {'replication_caught_up': [29, 30], 'source_writes_stopped': False, 'dependency_ready': [True, False], 'capacity_headroom': [6, 5, 10], 'cutover_generation': [7, 7]}
keys = list(invalid)
pair = {keys[N % len(keys)]: invalid[keys[N % len(keys)]], keys[(N+1) % len(keys)]: invalid[keys[(N+1) % len(keys)]]}
check('two independent violations in variant', solve(dict(r, **pair)), False)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
valid operation	`True`	`True`	Passed
Cutover loses writes beyond the destination applied watermark	`False`	`False`	Passed
Both regions accept writes during a single-writer cutover	`False`	`False`	Passed
Application traffic arrives before regional dependencies are ready	`False`	`False`	Passed
Destination capacity cannot serve the transferred load	`False`	`False`	Passed
A delayed cutover command reverses a newer regional decision	`True`	`False`	Failed
unrelated tracing metadata	`True`	`True`	Passed
repeat validation is pure	`True`	`True`	Passed
two independent violations in variant	`False`	`False`	Passed

SHA-256 / 9a7041798c3a947f0f4fe4627cb4498bd33ea4e02131a54686e02e82ca6b0c93

3 / The verified repair

Exit 0

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(r):
    return (r['replication_caught_up'][0] >= r['replication_caught_up'][1]) and (r['source_writes_stopped'] is True) and (all(r['dependency_ready'])) and (r['capacity_headroom'][0]+r['capacity_headroom'][1] <= r['capacity_headroom'][2]) and (r['cutover_generation'][0] > r['cutover_generation'][1])
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
r = {'replication_caught_up': [30, 30], 'source_writes_stopped': True, 'dependency_ready': [True, True], 'capacity_headroom': [4, 5, 10], 'cutover_generation': [8, 7]}
check('valid operation', solve(r), True)
check('Cutover loses writes beyond the destination applied watermark', solve(dict(r, **{'replication_caught_up': [29, 30]})), False)
check('Both regions accept writes during a single-writer cutover', solve(dict(r, **{'source_writes_stopped': False})), False)
check('Application traffic arrives before regional dependencies are ready', solve(dict(r, **{'dependency_ready': [True, False]})), False)
check('Destination capacity cannot serve the transferred load', solve(dict(r, **{'capacity_headroom': [6, 5, 10]})), False)
check('A delayed cutover command reverses a newer regional decision', solve(dict(r, **{'cutover_generation': [7, 7]})), False)
check('unrelated tracing metadata', solve(dict(r, trace='run-'+str(N))), True)
check('repeat validation is pure', solve(r), True)
invalid = {'replication_caught_up': [29, 30], 'source_writes_stopped': False, 'dependency_ready': [True, False], 'capacity_headroom': [6, 5, 10], 'cutover_generation': [7, 7]}
keys = list(invalid)
pair = {keys[N % len(keys)]: invalid[keys[N % len(keys)]], keys[(N+1) % len(keys)]: invalid[keys[(N+1) % len(keys)]]}
check('two independent violations in variant', solve(dict(r, **pair)), False)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
valid operation	`True`	`True`	Passed
Cutover loses writes beyond the destination applied watermark	`False`	`False`	Passed
Both regions accept writes during a single-writer cutover	`False`	`False`	Passed
Application traffic arrives before regional dependencies are ready	`False`	`False`	Passed
Destination capacity cannot serve the transferred load	`False`	`False`	Passed
A delayed cutover command reverses a newer regional decision	`False`	`False`	Passed
unrelated tracing metadata	`True`	`True`	Passed
repeat validation is pure	`True`	`True`	Passed
two independent violations in variant	`False`	`False`	Passed

SHA-256 / 63d3ae437385c6647c5da1ed7936d71501c6ba47c5a7a73f6502f94452cec68d

Verification & scope

This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:37:03.746387+00:00.

Case digest / 154e406ecb106a9dc71f911b3421ed795ba2bd40d2314d968de9027e2733d7fa