FA-99971 / Evaluation metric aggregation / Open access

Macro specificity is pooled across classes · case 01

Macro specificity is pooled across classes.

Verified by executionVariant 1 · 6 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

One-vs-rest true negatives are aggregated before division, allowing common classes to dominate.

THE FAILURE

One-vs-rest true negatives are aggregated before division, allowing common classes to dominate.

Unsuccessful approach: Averaging the class confusion counts before division keeps the support imbalance.

Case contract

For each supplied label compute TN/(TN+FP) in a one-vs-rest view; average equally. A label with no actual negatives contributes 0; an empty label list returns 0.

Why this case matters

A small explicit evaluation contract makes aggregation behavior reproducible across labeled examples.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(truth, pred, labels):
    tn=sum(a not in labels and b not in labels for a,b in zip(truth,pred)); fp=sum(a not in labels and b in labels for a,b in zip(truth,pred)); return tn/(tn+fp) if tn+fp else 0
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('class average',solve([0,0,0,1],[0,1,1,1],[0,1]),2/3)
check('all correct two classes',solve([0,1],[0,1],[0,1]),1)
check('empty labels',solve([0],[0],[]),0)
check('empty examples',solve([],[],[0,1]),0)
check('absent class has negative support',solve([0,0],[0,1],[0,1]),.25)
check('only one requested class',solve([N,0],[N,N],[N]),0)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
class average	`0`	`0.6666666666666666`	Failed
all correct two classes	`0`	`1`	Failed
empty labels	`1.0`	`0`	Failed
empty examples	`0`	`0`	Passed
absent class has negative support	`0`	`0.25`	Failed
only one requested class	`0.0`	`0`	Passed

SHA-256 / a0605357367bf5d7c398d56b1c62f33ae27376b10e956f00ff19d45a56d6cc48

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(truth, pred, labels):
    vals=[]
    for k in labels:
        tn=sum(a!=k and b!=k for a,b in zip(truth,pred)); fp=sum(a!=k and b==k for a,b in zip(truth,pred)); vals.append(tn/(tn+fp+1) if tn+fp else 0)
    return sum(vals)/len(vals) if vals else 0
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('class average',solve([0,0,0,1],[0,1,1,1],[0,1]),2/3)
check('all correct two classes',solve([0,1],[0,1],[0,1]),1)
check('empty labels',solve([0],[0],[]),0)
check('empty examples',solve([],[],[0,1]),0)
check('absent class has negative support',solve([0,0],[0,1],[0,1]),.25)
check('only one requested class',solve([N,0],[N,N],[N]),0)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
class average	`0.375`	`0.6666666666666666`	Failed
all correct two classes	`0.5`	`1`	Failed
empty labels	`0`	`0`	Passed
empty examples	`0.0`	`0`	Passed
absent class has negative support	`0.16666666666666666`	`0.25`	Failed
only one requested class	`0.0`	`0`	Passed

SHA-256 / dffe3860391b4cbc3ff84ec2b135d8791be3124c31037c27dfd6f4ab9b07fd96

HELD IN THE MEMBER ARCHIVE

The verified repair and its recorded checks are member-only.

This mechanism has 6 recorded checks per implementation. The open-access tier publishes the failure and the unsuccessful fix; the repaired source that passes every check, and the observations that prove it, are available to members.

Every case sharing this mechanism uses the same contract and the same repair, so this one record is held back for all of them.

Member access is invitation-based. Sign in with your invited account to inspect the repair.

Verification & scope

Inputs are finite toy labels and scores; this defines no benchmark protocol, production metric library, or model quality claim. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-30T16:48:30.355952+00:00.

Case digest / 52245c4595f93588b9866c0b04f6a1ebb9f957bd67a84f82998518c45c5c3351