FA-100011 / Evaluation metric aggregation / Open access

Sample weights are applied only to correct rows · case 01

Sample weights are applied only to correct rows.

Verified by executionVariant 1 · 6 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

Weighted correctness is normalized by row count rather than total sample weight.

THE FAILURE

Weighted correctness is normalized by row count rather than total sample weight.

Unsuccessful approach: Using only correct-row weights in the denominator forces every nonzero score to one.

Case contract

Return weighted accuracy for aligned labels and nonnegative weights; zero total weight yields 0.

Why this case matters

A small explicit evaluation contract makes aggregation behavior reproducible across labeled examples.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(truth, pred, weights):
    return sum(w for a,b,w in zip(truth,pred,weights) if a==b)/len(truth) if truth else 0
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('unequal weights',solve([1,0],[1,1],[N,1]),N/(N+1))
check('all right',solve([1,0],[1,0],[2,3]),1)
check('all wrong',solve([1],[0],[N]),0)
check('zero weights',solve([1],[1],[0]),0)
check('empty',solve([],[],[]),0)
check('weighted miss dominates',solve([1,0],[0,0],[N,1]),1/(N+1))
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
unequal weights	`0.5`	`0.5`	Passed
all right	`2.5`	`1`	Failed
all wrong	`0.0`	`0`	Passed
zero weights	`0.0`	`0`	Passed
empty	`0`	`0`	Passed
weighted miss dominates	`0.5`	`0.5`	Passed

SHA-256 / 89143b0ba799e63f0e4dbdb3ca1f0be0f3d64f74022880b5c035e44a09dff95a

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(truth, pred, weights):
    total=sum(weights); correct=sum(w for a,b,w in zip(truth,pred,weights) if a==b); return correct/(correct or 1) if total else 0
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('unequal weights',solve([1,0],[1,1],[N,1]),N/(N+1))
check('all right',solve([1,0],[1,0],[2,3]),1)
check('all wrong',solve([1],[0],[N]),0)
check('zero weights',solve([1],[1],[0]),0)
check('empty',solve([],[],[]),0)
check('weighted miss dominates',solve([1,0],[0,0],[N,1]),1/(N+1))
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
unequal weights	`1.0`	`0.5`	Failed
all right	`1.0`	`1`	Passed
all wrong	`0.0`	`0`	Passed
zero weights	`0`	`0`	Passed
empty	`0`	`0`	Passed
weighted miss dominates	`1.0`	`0.5`	Failed

SHA-256 / 7f29e08cbe2e1219e96704645489db1135b1df6e053ad8aeeeaeedced82901b5

HELD IN THE MEMBER ARCHIVE

The verified repair and its recorded checks are member-only.

This mechanism has 6 recorded checks per implementation. The open-access tier publishes the failure and the unsuccessful fix; the repaired source that passes every check, and the observations that prove it, are available to members.

Every case sharing this mechanism uses the same contract and the same repair, so this one record is held back for all of them.

Member access is invitation-based. Sign in with your invited account to inspect the repair.

Verification & scope

Inputs are finite toy labels and scores; this defines no benchmark protocol, production metric library, or model quality claim. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-30T16:48:30.810521+00:00.

Case digest / 6a1ac2e61f8ddf3187d1bf1d053b2c61bb19670e545e2805b0b293c52627c5de