FA-1076 / Reliability / Open access

Promote a standby after primary failure: Promotion permits two writable primaries · case 01

The failover promote operation is admitted even though promotion permits two writable primaries.

Verified by executionVariant 1 · 9 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The admission path omits the primary fenced invariant while validating the other operation preconditions.

VERIFIED REPAIR

Require r['primary_fenced'] is True together with every other stated precondition before accepting the operation.

Unsuccessful approach: Adding the primary fenced check repairs the reported defect, but replacing the adjacent rpo bound check loses that independent invariant.

Case contract

Return a Boolean admission decision for promote a standby after primary failure. The record r must satisfy all of: r['primary_fenced'] is True; r['rpo_bound'][0]-r['rpo_bound'][1] <= r['rpo_bound'][2]; r['standby_writable'] is True; r['promotion_epoch'][0] > r['promotion_epoch'][1]; r['routing_switched'][0] == r['routing_switched'][1]. Extra tracing fields are ignored; validation does not mutate the record.

Why this case matters

A deterministic local contract for reliability. Each negative fixture violates exactly one invariant. No transport timing, persistence, cryptographic verification, or full protocol implementation is claimed.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(r):
    return (r['rpo_bound'][0]-r['rpo_bound'][1] <= r['rpo_bound'][2]) and (r['standby_writable'] is True) and (r['promotion_epoch'][0] > r['promotion_epoch'][1]) and (r['routing_switched'][0] == r['routing_switched'][1])
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
r = {'primary_fenced': True, 'rpo_bound': [10, 9, 1], 'standby_writable': True, 'promotion_epoch': [6, 5], 'routing_switched': ['new', 'new']}
check('valid operation', solve(r), True)
check('Promotion permits two writable primaries', solve(dict(r, **{'primary_fenced': False})), False)
check('Standby lag exceeds the declared data-loss allowance', solve(dict(r, **{'rpo_bound': [10, 8, 1]})), False)
check('A read-only standby is advertised as write primary', solve(dict(r, **{'standby_writable': False})), False)
check('A delayed promotion reverses a newer failover', solve(dict(r, **{'promotion_epoch': [5, 5]})), False)
check('Promotion succeeds while writes still route to the old primary', solve(dict(r, **{'routing_switched': ['old', 'new']})), False)
check('unrelated tracing metadata', solve(dict(r, trace='run-'+str(N))), True)
check('repeat validation is pure', solve(r), True)
invalid = {'primary_fenced': False, 'rpo_bound': [10, 8, 1], 'standby_writable': False, 'promotion_epoch': [5, 5], 'routing_switched': ['old', 'new']}
keys = list(invalid)
pair = {keys[N % len(keys)]: invalid[keys[N % len(keys)]], keys[(N+1) % len(keys)]: invalid[keys[(N+1) % len(keys)]]}
check('two independent violations in variant', solve(dict(r, **pair)), False)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
valid operation	`True`	`True`	Passed
Promotion permits two writable primaries	`True`	`False`	Failed
Standby lag exceeds the declared data-loss allowance	`False`	`False`	Passed
A read-only standby is advertised as write primary	`False`	`False`	Passed
A delayed promotion reverses a newer failover	`False`	`False`	Passed
Promotion succeeds while writes still route to the old primary	`False`	`False`	Passed
unrelated tracing metadata	`True`	`True`	Passed
repeat validation is pure	`True`	`True`	Passed
two independent violations in variant	`False`	`False`	Passed

SHA-256 / 918993a781a4f67df8f08472d6712aa3210efc608524bdcf8199fd654b237dbd

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(r):
    return (r['primary_fenced'] is True) and (r['standby_writable'] is True) and (r['promotion_epoch'][0] > r['promotion_epoch'][1]) and (r['routing_switched'][0] == r['routing_switched'][1])
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
r = {'primary_fenced': True, 'rpo_bound': [10, 9, 1], 'standby_writable': True, 'promotion_epoch': [6, 5], 'routing_switched': ['new', 'new']}
check('valid operation', solve(r), True)
check('Promotion permits two writable primaries', solve(dict(r, **{'primary_fenced': False})), False)
check('Standby lag exceeds the declared data-loss allowance', solve(dict(r, **{'rpo_bound': [10, 8, 1]})), False)
check('A read-only standby is advertised as write primary', solve(dict(r, **{'standby_writable': False})), False)
check('A delayed promotion reverses a newer failover', solve(dict(r, **{'promotion_epoch': [5, 5]})), False)
check('Promotion succeeds while writes still route to the old primary', solve(dict(r, **{'routing_switched': ['old', 'new']})), False)
check('unrelated tracing metadata', solve(dict(r, trace='run-'+str(N))), True)
check('repeat validation is pure', solve(r), True)
invalid = {'primary_fenced': False, 'rpo_bound': [10, 8, 1], 'standby_writable': False, 'promotion_epoch': [5, 5], 'routing_switched': ['old', 'new']}
keys = list(invalid)
pair = {keys[N % len(keys)]: invalid[keys[N % len(keys)]], keys[(N+1) % len(keys)]: invalid[keys[(N+1) % len(keys)]]}
check('two independent violations in variant', solve(dict(r, **pair)), False)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
valid operation	`True`	`True`	Passed
Promotion permits two writable primaries	`False`	`False`	Passed
Standby lag exceeds the declared data-loss allowance	`True`	`False`	Failed
A read-only standby is advertised as write primary	`False`	`False`	Passed
A delayed promotion reverses a newer failover	`False`	`False`	Passed
Promotion succeeds while writes still route to the old primary	`False`	`False`	Passed
unrelated tracing metadata	`True`	`True`	Passed
repeat validation is pure	`True`	`True`	Passed
two independent violations in variant	`False`	`False`	Passed

SHA-256 / 8915cb81da94c73b6e40a7279dc852f937f86c9103eed2c1b302f1052247682b

3 / The verified repair

Exit 0

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(r):
    return (r['primary_fenced'] is True) and (r['rpo_bound'][0]-r['rpo_bound'][1] <= r['rpo_bound'][2]) and (r['standby_writable'] is True) and (r['promotion_epoch'][0] > r['promotion_epoch'][1]) and (r['routing_switched'][0] == r['routing_switched'][1])
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
r = {'primary_fenced': True, 'rpo_bound': [10, 9, 1], 'standby_writable': True, 'promotion_epoch': [6, 5], 'routing_switched': ['new', 'new']}
check('valid operation', solve(r), True)
check('Promotion permits two writable primaries', solve(dict(r, **{'primary_fenced': False})), False)
check('Standby lag exceeds the declared data-loss allowance', solve(dict(r, **{'rpo_bound': [10, 8, 1]})), False)
check('A read-only standby is advertised as write primary', solve(dict(r, **{'standby_writable': False})), False)
check('A delayed promotion reverses a newer failover', solve(dict(r, **{'promotion_epoch': [5, 5]})), False)
check('Promotion succeeds while writes still route to the old primary', solve(dict(r, **{'routing_switched': ['old', 'new']})), False)
check('unrelated tracing metadata', solve(dict(r, trace='run-'+str(N))), True)
check('repeat validation is pure', solve(r), True)
invalid = {'primary_fenced': False, 'rpo_bound': [10, 8, 1], 'standby_writable': False, 'promotion_epoch': [5, 5], 'routing_switched': ['old', 'new']}
keys = list(invalid)
pair = {keys[N % len(keys)]: invalid[keys[N % len(keys)]], keys[(N+1) % len(keys)]: invalid[keys[(N+1) % len(keys)]]}
check('two independent violations in variant', solve(dict(r, **pair)), False)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
valid operation	`True`	`True`	Passed
Promotion permits two writable primaries	`False`	`False`	Passed
Standby lag exceeds the declared data-loss allowance	`False`	`False`	Passed
A read-only standby is advertised as write primary	`False`	`False`	Passed
A delayed promotion reverses a newer failover	`False`	`False`	Passed
Promotion succeeds while writes still route to the old primary	`False`	`False`	Passed
unrelated tracing metadata	`True`	`True`	Passed
repeat validation is pure	`True`	`True`	Passed
two independent violations in variant	`False`	`False`	Passed

SHA-256 / 4d5271ee2d95b6384e0ebb8d873ed589261c47bd05cf7abcb05be97b66331b75

Verification & scope

This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:36:58.529650+00:00.

Case digest / 4cc6dd06a57867944eef72bf56b63145b5764719a91a39049cef6f903778df98