FA-081 / Distributed coordination / Open access

Fanout multiplies a parent request's retry allowance · case 01

Every child RPC spends the entire retry budget independently.

Verified by executionVariant 1 · 7 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The budget is copied to children instead of accounted for across the request tree.

VERIFIED REPAIR

Consume a single shared allowance with deterministic round-robin grants to pending branches.

Unsuccessful approach: Equal static partitions waste remainders and cannot transfer unused allowance from idle children.

Case contract

Given a nonnegative total allowance and nonnegative per-branch demand, return grants in branch order. Grant one unit per nonempty branch per round until total or all demand is exhausted. This models serialized admission to an already atomic shared counter.

Why this case matters

Models bounded retry amplification in scatter/gather RPC graphs; it does not attempt to simulate network timeouts or replace an atomic distributed budget store.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(total, demands):
    return [min(total, demand) for demand in demands]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('one shared allowance', solve(N, [N, N]), [(N+1)//2, N//2])
check('idle branch yields allowance', solve(N, [0, N]), [0, N])
check('leftover unit stays useful', solve(2*N+1, [N+1, N+1]), [N+1, N])
check('demand lower than allowance', solve(10*N, [N, 1]), [N, 1])
check('zero allowance', solve(0, [N, N]), [0, 0])
check('no branches', solve(N, []), [])
check('all branches idle', solve(N, [0, 0]), [0, 0])
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
one shared allowance	`[1, 1]`	`[1, 0]`	Failed
idle branch yields allowance	`[0, 1]`	`[0, 1]`	Passed
leftover unit stays useful	`[2, 2]`	`[2, 1]`	Failed
demand lower than allowance	`[1, 1]`	`[1, 1]`	Passed
zero allowance	`[0, 0]`	`[0, 0]`	Passed
no branches	`[]`	`[]`	Passed
all branches idle	`[0, 0]`	`[0, 0]`	Passed

SHA-256 / bc40db986c1f96acbaac70eb8b3fca8b33eea1e17877599aa0b73d8b349823a0

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(total, demands):
    share = total//len(demands) if demands else 0
    return [min(share, demand) for demand in demands]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('one shared allowance', solve(N, [N, N]), [(N+1)//2, N//2])
check('idle branch yields allowance', solve(N, [0, N]), [0, N])
check('leftover unit stays useful', solve(2*N+1, [N+1, N+1]), [N+1, N])
check('demand lower than allowance', solve(10*N, [N, 1]), [N, 1])
check('zero allowance', solve(0, [N, N]), [0, 0])
check('no branches', solve(N, []), [])
check('all branches idle', solve(N, [0, 0]), [0, 0])
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
one shared allowance	`[0, 0]`	`[1, 0]`	Failed
idle branch yields allowance	`[0, 0]`	`[0, 1]`	Failed
leftover unit stays useful	`[1, 1]`	`[2, 1]`	Failed
demand lower than allowance	`[1, 1]`	`[1, 1]`	Passed
zero allowance	`[0, 0]`	`[0, 0]`	Passed
no branches	`[]`	`[]`	Passed
all branches idle	`[0, 0]`	`[0, 0]`	Passed

SHA-256 / 3d598bbeaef8475865d77ff3e23c571d562f25d28f6d9d3a31b93becc3ae6368

3 / The verified repair

Exit 0

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(total, demands):
    grants = [0]*len(demands)
    while total > 0:
        progressed = False
        for i, demand in enumerate(demands):
            if grants[i] < demand and total:
                grants[i] += 1
                total -= 1
                progressed = True
        if not progressed:
            break
    return grants
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('one shared allowance', solve(N, [N, N]), [(N+1)//2, N//2])
check('idle branch yields allowance', solve(N, [0, N]), [0, N])
check('leftover unit stays useful', solve(2*N+1, [N+1, N+1]), [N+1, N])
check('demand lower than allowance', solve(10*N, [N, 1]), [N, 1])
check('zero allowance', solve(0, [N, N]), [0, 0])
check('no branches', solve(N, []), [])
check('all branches idle', solve(N, [0, 0]), [0, 0])
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
one shared allowance	`[1, 0]`	`[1, 0]`	Passed
idle branch yields allowance	`[0, 1]`	`[0, 1]`	Passed
leftover unit stays useful	`[2, 1]`	`[2, 1]`	Passed
demand lower than allowance	`[1, 1]`	`[1, 1]`	Passed
zero allowance	`[0, 0]`	`[0, 0]`	Passed
no branches	`[]`	`[]`	Passed
all branches idle	`[0, 0]`	`[0, 0]`	Passed

SHA-256 / 6e604caeddfa7b2869f4b2eb54ee6e53f4a91aef40b4cd77efed53d1d59f1bcb

Verification & scope

This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:36:50.427483+00:00.

Case digest / fa64d1c41195d91d8253d74f6371e287ed2294966993ee369757172829052ea3