FA-666 / Replication / Open access

Retire a replica and discard its data: Active snapshot readers lose their pinned source · case 01

The replica removal operation is admitted even though active snapshot readers lose their pinned source.

Verified by executionVariant 1 · 9 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The admission path omits the reader drain invariant while validating the other operation preconditions.

VERIFIED REPAIR

Require r['reader_drain'] == 0 together with every other stated precondition before accepting the operation.

Unsuccessful approach: Adding the reader drain check repairs the reported defect, but replacing the adjacent membership commit check loses that independent invariant.

Case contract

Return a Boolean admission decision for retire a replica and discard its data. The record r must satisfy all of: r['replacement_count'][0]-1 >= r['replacement_count'][1]; r['replacement_durable'] is True; len(r['handoff_complete']) == 0; r['reader_drain'] == 0; r['membership_commit'][0] <= r['membership_commit'][1]. Extra tracing fields are ignored; validation does not mutate the record.

Why this case matters

A deterministic local contract for replication. Each negative fixture violates exactly one invariant. No transport timing, persistence, cryptographic verification, or full protocol implementation is claimed.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(r):
    return (r['replacement_count'][0]-1 >= r['replacement_count'][1]) and (r['replacement_durable'] is True) and (len(r['handoff_complete']) == 0) and (r['membership_commit'][0] <= r['membership_commit'][1])
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
r = {'replacement_count': [4, 3], 'replacement_durable': True, 'handoff_complete': [], 'reader_drain': 0, 'membership_commit': [20, 20]}
check('valid operation', solve(r), True)
check('Retirement leaves fewer copies than the required factor', solve(dict(r, **{'replacement_count': [3, 3]})), False)
check('The replacement has only volatile copied state', solve(dict(r, **{'replacement_durable': False})), False)
check('Keys still assigned to the retiring replica are lost', solve(dict(r, **{'handoff_complete': ['key-7']})), False)
check('Active snapshot readers lose their pinned source', solve(dict(r, **{'reader_drain': 2})), False)
check('Physical removal precedes committed membership removal', solve(dict(r, **{'membership_commit': [21, 20]})), False)
check('unrelated tracing metadata', solve(dict(r, trace='run-'+str(N))), True)
check('repeat validation is pure', solve(r), True)
invalid = {'replacement_count': [3, 3], 'replacement_durable': False, 'handoff_complete': ['key-7'], 'reader_drain': 2, 'membership_commit': [21, 20]}
keys = list(invalid)
pair = {keys[N % len(keys)]: invalid[keys[N % len(keys)]], keys[(N+1) % len(keys)]: invalid[keys[(N+1) % len(keys)]]}
check('two independent violations in variant', solve(dict(r, **pair)), False)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
valid operation	`True`	`True`	Passed
Retirement leaves fewer copies than the required factor	`False`	`False`	Passed
The replacement has only volatile copied state	`False`	`False`	Passed
Keys still assigned to the retiring replica are lost	`False`	`False`	Passed
Active snapshot readers lose their pinned source	`True`	`False`	Failed
Physical removal precedes committed membership removal	`False`	`False`	Passed
unrelated tracing metadata	`True`	`True`	Passed
repeat validation is pure	`True`	`True`	Passed
two independent violations in variant	`False`	`False`	Passed

SHA-256 / ec02e7b8e9ec40b2b32e56d609e82733d2d636659adbb34eff05b0ad9aaae670

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(r):
    return (r['replacement_count'][0]-1 >= r['replacement_count'][1]) and (r['replacement_durable'] is True) and (len(r['handoff_complete']) == 0) and (r['reader_drain'] == 0)
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
r = {'replacement_count': [4, 3], 'replacement_durable': True, 'handoff_complete': [], 'reader_drain': 0, 'membership_commit': [20, 20]}
check('valid operation', solve(r), True)
check('Retirement leaves fewer copies than the required factor', solve(dict(r, **{'replacement_count': [3, 3]})), False)
check('The replacement has only volatile copied state', solve(dict(r, **{'replacement_durable': False})), False)
check('Keys still assigned to the retiring replica are lost', solve(dict(r, **{'handoff_complete': ['key-7']})), False)
check('Active snapshot readers lose their pinned source', solve(dict(r, **{'reader_drain': 2})), False)
check('Physical removal precedes committed membership removal', solve(dict(r, **{'membership_commit': [21, 20]})), False)
check('unrelated tracing metadata', solve(dict(r, trace='run-'+str(N))), True)
check('repeat validation is pure', solve(r), True)
invalid = {'replacement_count': [3, 3], 'replacement_durable': False, 'handoff_complete': ['key-7'], 'reader_drain': 2, 'membership_commit': [21, 20]}
keys = list(invalid)
pair = {keys[N % len(keys)]: invalid[keys[N % len(keys)]], keys[(N+1) % len(keys)]: invalid[keys[(N+1) % len(keys)]]}
check('two independent violations in variant', solve(dict(r, **pair)), False)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
valid operation	`True`	`True`	Passed
Retirement leaves fewer copies than the required factor	`False`	`False`	Passed
The replacement has only volatile copied state	`False`	`False`	Passed
Keys still assigned to the retiring replica are lost	`False`	`False`	Passed
Active snapshot readers lose their pinned source	`False`	`False`	Passed
Physical removal precedes committed membership removal	`True`	`False`	Failed
unrelated tracing metadata	`True`	`True`	Passed
repeat validation is pure	`True`	`True`	Passed
two independent violations in variant	`False`	`False`	Passed

SHA-256 / 879993aa1ec426c1d6ee3744ef82b1a5368b61c94f724d653c7a2790a050ad44

3 / The verified repair

Exit 0

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(r):
    return (r['replacement_count'][0]-1 >= r['replacement_count'][1]) and (r['replacement_durable'] is True) and (len(r['handoff_complete']) == 0) and (r['reader_drain'] == 0) and (r['membership_commit'][0] <= r['membership_commit'][1])
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
r = {'replacement_count': [4, 3], 'replacement_durable': True, 'handoff_complete': [], 'reader_drain': 0, 'membership_commit': [20, 20]}
check('valid operation', solve(r), True)
check('Retirement leaves fewer copies than the required factor', solve(dict(r, **{'replacement_count': [3, 3]})), False)
check('The replacement has only volatile copied state', solve(dict(r, **{'replacement_durable': False})), False)
check('Keys still assigned to the retiring replica are lost', solve(dict(r, **{'handoff_complete': ['key-7']})), False)
check('Active snapshot readers lose their pinned source', solve(dict(r, **{'reader_drain': 2})), False)
check('Physical removal precedes committed membership removal', solve(dict(r, **{'membership_commit': [21, 20]})), False)
check('unrelated tracing metadata', solve(dict(r, trace='run-'+str(N))), True)
check('repeat validation is pure', solve(r), True)
invalid = {'replacement_count': [3, 3], 'replacement_durable': False, 'handoff_complete': ['key-7'], 'reader_drain': 2, 'membership_commit': [21, 20]}
keys = list(invalid)
pair = {keys[N % len(keys)]: invalid[keys[N % len(keys)]], keys[(N+1) % len(keys)]: invalid[keys[(N+1) % len(keys)]]}
check('two independent violations in variant', solve(dict(r, **pair)), False)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
valid operation	`True`	`True`	Passed
Retirement leaves fewer copies than the required factor	`False`	`False`	Passed
The replacement has only volatile copied state	`False`	`False`	Passed
Keys still assigned to the retiring replica are lost	`False`	`False`	Passed
Active snapshot readers lose their pinned source	`False`	`False`	Passed
Physical removal precedes committed membership removal	`False`	`False`	Passed
unrelated tracing metadata	`True`	`True`	Passed
repeat validation is pure	`True`	`True`	Passed
two independent violations in variant	`False`	`False`	Passed

SHA-256 / 26574c4899d4ab75de3ee303353f85cd810d03389d7872c2571aaddf0af37020

Verification & scope

This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:36:54.973552+00:00.

Case digest / 2d5b7788b6ac6df0a5b620be36e89c152602bf46cf0d5b66044a71209a7bc1ff