FA-99321 / Distributed training aggregation / Open access
Worker variances omit between-worker spread · case 01
Worker variances omit between-worker spread.
ROOT CAUSE
Averaging worker-local variances discards differences between their means.
THE FAILURE
Averaging worker-local variances discards differences between their means.
Unsuccessful approach: Weighting local variances by worker count still omits the between-worker mean contribution.
Case contract
Each worker supplies [count,sum,sumsq]; return the global population variance rounded to 12 decimals, or None when total count is zero.
Why this case matters
A toy distributed-training contract makes replica and shard contribution rules explicit without claiming framework compatibility or performance.
1 / The failure
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(stats):
return sum(q/c-(s/c)**2 for c,s,q in stats if c)/sum(1 for c,s,q in stats if c) if any(c for c,s,q in stats) else None
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('between-worker spread',solve([[1,0,0],[1,2,4]]),1)
check('unequal workers with within variance',solve([[2,2*N,4*N*N],[1,5*N,25*N*N]]),round(38*N*N/9,12))
check('one constant worker',solve([[N,3*N,9*N]]),0)
check('empty',solve([]),None)
check('zero observations',solve([[0,0,0]]),None)
check('zero worker excluded',solve([[0,99*N,999*N],[1,N,N*N]]),0)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| between-worker spread | 0.0 | 1 | Failed |
| unequal workers with within variance | 0.5 | 4.222222222222 | Failed |
| one constant worker | 0.0 | 0 | Passed |
| empty | None | None | Passed |
| zero observations | None | None | Passed |
| zero worker excluded | 0.0 | 0 | Passed |
SHA-256 / 0624690b97f6c823a0f7ca71106e9651425762979aef416e3e29c937eb0d30bd
2 / The unsuccessful fix
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(stats):
n=sum(c for c,s,q in stats); return sum((q/c-(s/c)**2)*c for c,s,q in stats if c)/n if n else None
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('between-worker spread',solve([[1,0,0],[1,2,4]]),1)
check('unequal workers with within variance',solve([[2,2*N,4*N*N],[1,5*N,25*N*N]]),round(38*N*N/9,12))
check('one constant worker',solve([[N,3*N,9*N]]),0)
check('empty',solve([]),None)
check('zero observations',solve([[0,0,0]]),None)
check('zero worker excluded',solve([[0,99*N,999*N],[1,N,N*N]]),0)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| between-worker spread | 0.0 | 1 | Failed |
| unequal workers with within variance | 0.6666666666666666 | 4.222222222222 | Failed |
| one constant worker | 0.0 | 0 | Passed |
| empty | None | None | Passed |
| zero observations | None | None | Passed |
| zero worker excluded | 0.0 | 0 | Passed |
SHA-256 / a0c3e7827307832b5e5638f3744ac0f1734f5483c73ce6b205e7bb4b50ca22ae
HELD IN THE MEMBER ARCHIVE
The verified repair and its recorded checks are member-only.
This mechanism has 6 recorded checks per implementation. The open-access tier publishes the failure and the unsuccessful fix; the repaired source that passes every check, and the observations that prove it, are available to members.
Every case sharing this mechanism uses the same contract and the same repair, so this one record is held back for all of them.
Member access is invitation-based. Sign in with your invited account to inspect the repair.
Sign in to the archive ↗Verification & scope
Inputs are already materialized Python values from a fixed toy collective; no transport, failure recovery, optimizer state, or real model behavior is represented. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.
Observations recorded using Python 3.12.14 at 2026-09-30T16:48:23.078722+00:00.
Case digest / 898f1dd8f17ee02c57e6e7392a0c64fdd5a2be492a3cb4794076b88d0ca59fb9