FA-1041 / Reliability / Open access

Evict an unhealthy backend: Correlated failures evict more than the allowed pool fraction · case 01

The health eviction operation is admitted even though correlated failures evict more than the allowed pool fraction.

Verified by executionVariant 1 · 9 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The admission path omits the ejection limit invariant while validating the other operation preconditions.

VERIFIED REPAIR

Require (r['ejection_limit'][0]+1)*100 <= r['ejection_limit'][1]*r['ejection_limit'][2] together with every other stated precondition before accepting the operation.

Unsuccessful approach: Adding the ejection limit check repairs the reported defect, but replacing the adjacent backend incarnation check loses that independent invariant.

Case contract

Return a Boolean admission decision for evict an unhealthy backend. The record r must satisfy all of: r['consecutive_failures'][0] >= r['consecutive_failures'][1]; r['minimum_survivors'][0]-1 >= r['minimum_survivors'][1]; r['sample_freshness'][0]-r['sample_freshness'][1] <= r['sample_freshness'][2]; (r['ejection_limit'][0]+1)*100 <= r['ejection_limit'][1]*r['ejection_limit'][2]; r['backend_incarnation'][0] == r['backend_incarnation'][1]. Extra tracing fields are ignored; validation does not mutate the record.

Why this case matters

A deterministic local contract for reliability. Each negative fixture violates exactly one invariant. No transport timing, persistence, cryptographic verification, or full protocol implementation is claimed.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(r):
    return (r['consecutive_failures'][0] >= r['consecutive_failures'][1]) and (r['minimum_survivors'][0]-1 >= r['minimum_survivors'][1]) and (r['sample_freshness'][0]-r['sample_freshness'][1] <= r['sample_freshness'][2]) and (r['backend_incarnation'][0] == r['backend_incarnation'][1])
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
r = {'consecutive_failures': [3, 3], 'minimum_survivors': [3, 2], 'sample_freshness': [10, 8, 2], 'ejection_limit': [1, 10, 20], 'backend_incarnation': ['p2', 'p2']}
check('valid operation', solve(r), True)
check('One intermittent failure causes premature eviction', solve(dict(r, **{'consecutive_failures': [2, 3]})), False)
check('Eviction removes the last required healthy capacity', solve(dict(r, **{'minimum_survivors': [2, 2]})), False)
check('Old failure samples evict a recovered backend', solve(dict(r, **{'sample_freshness': [10, 7, 2]})), False)
check('Correlated failures evict more than the allowed pool fraction', solve(dict(r, **{'ejection_limit': [2, 10, 20]})), False)
check('A failed probe evicts a replacement process at the same address', solve(dict(r, **{'backend_incarnation': ['p1', 'p2']})), False)
check('unrelated tracing metadata', solve(dict(r, trace='run-'+str(N))), True)
check('repeat validation is pure', solve(r), True)
invalid = {'consecutive_failures': [2, 3], 'minimum_survivors': [2, 2], 'sample_freshness': [10, 7, 2], 'ejection_limit': [2, 10, 20], 'backend_incarnation': ['p1', 'p2']}
keys = list(invalid)
pair = {keys[N % len(keys)]: invalid[keys[N % len(keys)]], keys[(N+1) % len(keys)]: invalid[keys[(N+1) % len(keys)]]}
check('two independent violations in variant', solve(dict(r, **pair)), False)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
valid operation	`True`	`True`	Passed
One intermittent failure causes premature eviction	`False`	`False`	Passed
Eviction removes the last required healthy capacity	`False`	`False`	Passed
Old failure samples evict a recovered backend	`False`	`False`	Passed
Correlated failures evict more than the allowed pool fraction	`True`	`False`	Failed
A failed probe evicts a replacement process at the same address	`False`	`False`	Passed
unrelated tracing metadata	`True`	`True`	Passed
repeat validation is pure	`True`	`True`	Passed
two independent violations in variant	`False`	`False`	Passed

SHA-256 / 13547e3ad97bc8fdf0a4b501e81b907c88ce55a07010379e21fdceb71db53360

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(r):
    return (r['consecutive_failures'][0] >= r['consecutive_failures'][1]) and (r['minimum_survivors'][0]-1 >= r['minimum_survivors'][1]) and (r['sample_freshness'][0]-r['sample_freshness'][1] <= r['sample_freshness'][2]) and ((r['ejection_limit'][0]+1)*100 <= r['ejection_limit'][1]*r['ejection_limit'][2])
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
r = {'consecutive_failures': [3, 3], 'minimum_survivors': [3, 2], 'sample_freshness': [10, 8, 2], 'ejection_limit': [1, 10, 20], 'backend_incarnation': ['p2', 'p2']}
check('valid operation', solve(r), True)
check('One intermittent failure causes premature eviction', solve(dict(r, **{'consecutive_failures': [2, 3]})), False)
check('Eviction removes the last required healthy capacity', solve(dict(r, **{'minimum_survivors': [2, 2]})), False)
check('Old failure samples evict a recovered backend', solve(dict(r, **{'sample_freshness': [10, 7, 2]})), False)
check('Correlated failures evict more than the allowed pool fraction', solve(dict(r, **{'ejection_limit': [2, 10, 20]})), False)
check('A failed probe evicts a replacement process at the same address', solve(dict(r, **{'backend_incarnation': ['p1', 'p2']})), False)
check('unrelated tracing metadata', solve(dict(r, trace='run-'+str(N))), True)
check('repeat validation is pure', solve(r), True)
invalid = {'consecutive_failures': [2, 3], 'minimum_survivors': [2, 2], 'sample_freshness': [10, 7, 2], 'ejection_limit': [2, 10, 20], 'backend_incarnation': ['p1', 'p2']}
keys = list(invalid)
pair = {keys[N % len(keys)]: invalid[keys[N % len(keys)]], keys[(N+1) % len(keys)]: invalid[keys[(N+1) % len(keys)]]}
check('two independent violations in variant', solve(dict(r, **pair)), False)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
valid operation	`True`	`True`	Passed
One intermittent failure causes premature eviction	`False`	`False`	Passed
Eviction removes the last required healthy capacity	`False`	`False`	Passed
Old failure samples evict a recovered backend	`False`	`False`	Passed
Correlated failures evict more than the allowed pool fraction	`False`	`False`	Passed
A failed probe evicts a replacement process at the same address	`True`	`False`	Failed
unrelated tracing metadata	`True`	`True`	Passed
repeat validation is pure	`True`	`True`	Passed
two independent violations in variant	`False`	`False`	Passed

SHA-256 / 870fcf855fc1051c5df11ba52ca24a8015558e27a7a1c62443a870ba34a74149

3 / The verified repair

Exit 0

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(r):
    return (r['consecutive_failures'][0] >= r['consecutive_failures'][1]) and (r['minimum_survivors'][0]-1 >= r['minimum_survivors'][1]) and (r['sample_freshness'][0]-r['sample_freshness'][1] <= r['sample_freshness'][2]) and ((r['ejection_limit'][0]+1)*100 <= r['ejection_limit'][1]*r['ejection_limit'][2]) and (r['backend_incarnation'][0] == r['backend_incarnation'][1])
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
r = {'consecutive_failures': [3, 3], 'minimum_survivors': [3, 2], 'sample_freshness': [10, 8, 2], 'ejection_limit': [1, 10, 20], 'backend_incarnation': ['p2', 'p2']}
check('valid operation', solve(r), True)
check('One intermittent failure causes premature eviction', solve(dict(r, **{'consecutive_failures': [2, 3]})), False)
check('Eviction removes the last required healthy capacity', solve(dict(r, **{'minimum_survivors': [2, 2]})), False)
check('Old failure samples evict a recovered backend', solve(dict(r, **{'sample_freshness': [10, 7, 2]})), False)
check('Correlated failures evict more than the allowed pool fraction', solve(dict(r, **{'ejection_limit': [2, 10, 20]})), False)
check('A failed probe evicts a replacement process at the same address', solve(dict(r, **{'backend_incarnation': ['p1', 'p2']})), False)
check('unrelated tracing metadata', solve(dict(r, trace='run-'+str(N))), True)
check('repeat validation is pure', solve(r), True)
invalid = {'consecutive_failures': [2, 3], 'minimum_survivors': [2, 2], 'sample_freshness': [10, 7, 2], 'ejection_limit': [2, 10, 20], 'backend_incarnation': ['p1', 'p2']}
keys = list(invalid)
pair = {keys[N % len(keys)]: invalid[keys[N % len(keys)]], keys[(N+1) % len(keys)]: invalid[keys[(N+1) % len(keys)]]}
check('two independent violations in variant', solve(dict(r, **pair)), False)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
valid operation	`True`	`True`	Passed
One intermittent failure causes premature eviction	`False`	`False`	Passed
Eviction removes the last required healthy capacity	`False`	`False`	Passed
Old failure samples evict a recovered backend	`False`	`False`	Passed
Correlated failures evict more than the allowed pool fraction	`False`	`False`	Passed
A failed probe evicts a replacement process at the same address	`False`	`False`	Passed
unrelated tracing metadata	`True`	`True`	Passed
repeat validation is pure	`True`	`True`	Passed
two independent violations in variant	`False`	`False`	Passed

SHA-256 / b813b1b21004f3eb81a31aa1aa4a7d17977a8d643f38828702f6d0233b41534e

Verification & scope

This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:36:58.025774+00:00.

Case digest / b9c4a73a0152259231a72ec26817af33ed0a31f076f29cff4dee36b87923a8f4