FA-99266 / Distributed training aggregation / Open access

Gradient shards are clipped independently rather than by the global norm · case 01

Gradient shards are clipped independently rather than by the global norm.

Verified by executionVariant 1 · 7 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

Each rank computes a clipping factor from only its locally owned coordinates.

THE FAILURE

Each rank computes a clipping factor from only its locally owned coordinates.

Unsuccessful approach: Using the global root-mean-square magnitude instead of the L2 norm still underestimates the combined norm when coordinates span multiple shards.

Case contract

Given disjoint gradient shards and a nonnegative limit, return the same shard layout scaled by min(1, limit/global_L2_norm); zero norm uses scale one and results are rounded to 12 decimals.

Why this case matters

A toy distributed-training contract makes replica and shard contribution rules explicit without claiming framework compatibility or performance.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(shards, limit):
    out=[]
    for shard in shards:
        norm=sum(v*v for v in shard)**0.5
        scale=min(1,limit/norm) if norm else 1
        out.append([round(v*scale,12) for v in shard])
    return out
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('norm spans two shards',solve([[3*N,0],[0,4*N]],2*N),[[round(1.2*N,12),0],[0,round(1.6*N,12)]])
check('uneven shard widths',solve([[3*N],[4*N,0]],2*N),[[round(1.2*N,12)],[round(1.6*N,12),0]])
check('below limit remains unchanged',solve([[N],[N]],2*N),[[N],[N]])
check('single shard same global rule',solve([[3*N,4*N]],3*N),[[round(1.8*N,12),round(2.4*N,12)]])
check('zero norm',solve([[0,0],[]],N),[[0,0],[]])
check('zero limit',solve([[N]],0),[[0]])
check('empty shards',solve([],N),[])
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
norm spans two shards	`[[2.0, 0.0], [0.0, 2.0]]`	`[[1.2, 0], [0, 1.6]]`	Failed
uneven shard widths	`[[2.0], [2.0, 0.0]]`	`[[1.2], [1.6, 0]]`	Failed
below limit remains unchanged	`[[1], [1]]`	`[[1], [1]]`	Passed
single shard same global rule	`[[1.8, 2.4]]`	`[[1.8, 2.4]]`	Passed
zero norm	`[[0, 0], []]`	`[[0, 0], []]`	Passed
zero limit	`[[0.0]]`	`[[0]]`	Passed
empty shards	`[]`	`[]`	Passed

SHA-256 / 0976d76ff7cefd6d7129e98d2237d0171cc94d84009d2fc0c380ee7aaef79c71

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(shards, limit):
    total=sum(v*v for shard in shards for v in shard); count=sum(len(shard) for shard in shards)
    norm=(total/count)**0.5 if count else 0
    scale=min(1,limit/norm) if norm else 1
    return [[round(v*scale,12) for v in shard] for shard in shards]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('norm spans two shards',solve([[3*N,0],[0,4*N]],2*N),[[round(1.2*N,12),0],[0,round(1.6*N,12)]])
check('uneven shard widths',solve([[3*N],[4*N,0]],2*N),[[round(1.2*N,12)],[round(1.6*N,12),0]])
check('below limit remains unchanged',solve([[N],[N]],2*N),[[N],[N]])
check('single shard same global rule',solve([[3*N,4*N]],3*N),[[round(1.8*N,12),round(2.4*N,12)]])
check('zero norm',solve([[0,0],[]],N),[[0,0],[]])
check('zero limit',solve([[N]],0),[[0]])
check('empty shards',solve([],N),[])
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
norm spans two shards	`[[2.4, 0.0], [0.0, 3.2]]`	`[[1.2, 0], [0, 1.6]]`	Failed
uneven shard widths	`[[2.078460969083], [2.77128129211, 0.0]]`	`[[1.2], [1.6, 0]]`	Failed
below limit remains unchanged	`[[1], [1]]`	`[[1], [1]]`	Passed
single shard same global rule	`[[2.545584412272, 3.394112549695]]`	`[[1.8, 2.4]]`	Failed
zero norm	`[[0, 0], []]`	`[[0, 0], []]`	Passed
zero limit	`[[0.0]]`	`[[0]]`	Passed
empty shards	`[]`	`[]`	Passed

SHA-256 / ab79be2f71aab62cf1b615b1f2eaad633f8bbb937a890818810c8f0dae7116b6

HELD IN THE MEMBER ARCHIVE

The verified repair and its recorded checks are member-only.

This mechanism has 7 recorded checks per implementation. The open-access tier publishes the failure and the unsuccessful fix; the repaired source that passes every check, and the observations that prove it, are available to members.

Every case sharing this mechanism uses the same contract and the same repair, so this one record is held back for all of them.

Member access is invitation-based. Sign in with your invited account to inspect the repair.

Verification & scope

Inputs are already materialized Python values from a fixed toy collective; no transport, failure recovery, optimizer state, or real model behavior is represented. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-30T16:48:22.178536+00:00.

Case digest / 72c690d2cd7825d9b2073714700a8e7b21a076f535a0b9534ef607479305551f