FA-99306 / Distributed training aggregation / Open access

All-reduce silently truncates vectors with mismatched shapes · case 01

All-reduce silently truncates vectors with mismatched shapes.

Verified by executionVariant 1 · 6 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

Zip-based summation stops at the shortest worker vector and hides malformed contributions.

THE FAILURE

Zip-based summation stops at the shortest worker vector and hides malformed contributions.

Unsuccessful approach: Padding short vectors with zeros still accepts an invalid collective shape.

Case contract

Given worker vectors, return elementwise sum if all lengths match, otherwise None; empty input returns [].

Why this case matters

A toy distributed-training contract makes replica and shard contribution rules explicit without claiming framework compatibility or performance.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(parts):
    return [sum(col) for col in zip(*parts)] if parts else []
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('equal shape',solve([[N,2],[3*N,4]]),[4*N,6])
check('mismatch rejected',solve([[N,2],[3*N]]),None)
check('empty',solve([]),[])
check('singleton empty vector',solve([[]]),[])
check('singleton values',solve([[N]]),[N])
check('all empty shapes',solve([[],[]]),[])
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
equal shape	`[4, 6]`	`[4, 6]`	Passed
mismatch rejected	`[4]`	`None`	Failed
empty	`[]`	`[]`	Passed
singleton empty vector	`[]`	`[]`	Passed
singleton values	`[1]`	`[1]`	Passed
all empty shapes	`[]`	`[]`	Passed

SHA-256 / b34df758a386c986fec44e8a555796828acfade0bb876cdc11462e2ce976ad1d

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(parts):
    w=max(map(len,parts),default=0); return [sum(v[j] if j<len(v) else 0 for v in parts) for j in range(w)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('equal shape',solve([[N,2],[3*N,4]]),[4*N,6])
check('mismatch rejected',solve([[N,2],[3*N]]),None)
check('empty',solve([]),[])
check('singleton empty vector',solve([[]]),[])
check('singleton values',solve([[N]]),[N])
check('all empty shapes',solve([[],[]]),[])
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
equal shape	`[4, 6]`	`[4, 6]`	Passed
mismatch rejected	`[4, 2]`	`None`	Failed
empty	`[]`	`[]`	Passed
singleton empty vector	`[]`	`[]`	Passed
singleton values	`[1]`	`[1]`	Passed
all empty shapes	`[]`	`[]`	Passed

SHA-256 / 2371146cbbdcf6e24156500196090cf2ffd1035720b0515b460e616c42f7d58f

HELD IN THE MEMBER ARCHIVE

The verified repair and its recorded checks are member-only.

This mechanism has 6 recorded checks per implementation. The open-access tier publishes the failure and the unsuccessful fix; the repaired source that passes every check, and the observations that prove it, are available to members.

Every case sharing this mechanism uses the same contract and the same repair, so this one record is held back for all of them.

Member access is invitation-based. Sign in with your invited account to inspect the repair.

Verification & scope

Inputs are already materialized Python values from a fixed toy collective; no transport, failure recovery, optimizer state, or real model behavior is represented. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-30T16:48:22.633481+00:00.

Case digest / 7ff9736fec231603ffcf898154cccf5653f06d543440608aa65f79ea3fadde11