FAILURE MAP
← Case archive

FA-72191 / Error-correcting codes / Open access

Row retirement re-retires rows on later events · case 01

Retirement timestamps move later as more errors arrive on an already retired row.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

Events for rows that are already retired are not skipped.

VERIFIED REPAIR

Ignore every event for a row once it has been retired.

Unsuccessful approach: Skipping only corrected errors still lets a later uncorrectable error overwrite the timestamp.

Case contract

Memory-controller ECC policy. events are [time, row, kind] with kind "ce" (corrected) or "ue" (uncorrectable), processed in time order (stable for equal times). A "ue" retires its row immediately. A "ce" is added to the row's history, entries at or before time - window expire, and the row retires when the remaining count reaches the threshold. Events for a retired row are ignored. Return sorted [row, retirement time] pairs.

Why this case matters

Servers retire DRAM rows that show repeated corrected errors before they turn uncorrectable.

1 / The failure

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(events, window, threshold):
    retired = {}
    hist = {}
    for t, row, kind in sorted(events, key=lambda e: e[0]):
        if kind == 'ue':
            retired[row] = t
            continue
        h = [x for x in hist.get(row, []) if x > t - window] + [t]
        hist[row] = h
        if len(h) >= threshold:
            retired[row] = t
    return sorted([rw, tm] for rw, tm in retired.items())
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[['regression [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...', [[[21, 9, 'ce'], [27, 2, 'ue'], [0, 2, 'ue'], [21, 1, 'ce'], [9, 10, 'ce'], [0, 1, 'ce']], 5, 2], [[2, 0]]], ['regression [[[30, 1, "ce"], [30, 1, "ce"], [0, 3, "ce"], [16, 1, "ue"], [21, 3...', [[[30, 1, 'ce'], [30, 1, 'ce'], [0, 3, 'ce'], [16, 1, 'ue'], [21, 3, 'ue'], [33, 3, 'ce'], [30, 3, 'ue'], [29, 9, 'ue'], [1, 9, 'ce']], 15, 2], [[1, 16], [3, 21], [9, 29]]], ['control [[[39, 3, "ce"], [20, 10, "ce"], [16, 1, "ce"], [20, 10, "ce"], [22...', [[[39, 3, 'ce'], [20, 10, 'ce'], [16, 1, 'ce'], [20, 10, 'ce'], [22, 3, 'ue'], [40, 10, 'ce'], [15, 10, 'ce']], 5, 3], [[3, 22]]], ['control [[[14, 10, "ce"], [33, 1, "ce"], [3, 9, "ce"], [11, 1, "ce"], [9, 2...', [[[14, 10, 'ce'], [33, 1, 'ce'], [3, 9, 'ce'], [11, 1, 'ce'], [9, 2, 'ce'], [7, 9, 'ce'], [29, 3, 'ce'], [27, 1, 'ce']], 10, 3], []], ['control [[[35, 3, "ce"], [11, 1, "ce"], [8, 1, "ce"], [28, 3, "ce"], [3, 10...', [[[35, 3, 'ce'], [11, 1, 'ce'], [8, 1, 'ce'], [28, 3, 'ce'], [3, 10, 'ue'], [23, 3, 'ce'], [13, 10, 'ce']], 15, 3], [[3, 35], [10, 3]]], ['control [[[2, 10, "ue"], [35, 1, "ce"], [6, 3, "ue"], [37, 10, "ce"]], 10, 3]', [[[2, 10, 'ue'], [35, 1, 'ce'], [6, 3, 'ue'], [37, 10, 'ce']], 10, 3], [[3, 6], [10, 2]]], ['control [[[5, 9, "ce"], [3, 10, "ue"], [23, 1, "ce"]], 15, 3]', [[[5, 9, 'ce'], [3, 10, 'ue'], [23, 1, 'ce']], 15, 3], [[10, 3]]], ['control [[[38, 2, "ce"], [26, 10, "ce"], [5, 1, "ce"], [32, 2, "ce"]], 5, 3]', [[[38, 2, 'ce'], [26, 10, 'ce'], [5, 1, 'ce'], [32, 2, 'ce']], 5, 3], []]], [['regression [[[5, 2, "ue"], [6, 2, "ce"], [7, 2, "ue"]], 10, 3]', [[[5, 2, 'ue'], [6, 2, 'ce'], [7, 2, 'ue']], 10, 3], [[2, 5]]], ['regression [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...', [[[21, 9, 'ce'], [27, 2, 'ue'], [0, 2, 'ue'], [21, 1, 'ce'], [9, 10, 'ce'], [0, 1, 'ce']], 5, 2], [[2, 0]]], ['partial-repair [[[30, 1, "ce"], [30, 1, "ce"], [0, 3, "ce"], [16, 1, "ue"], [21, 3...', [[[30, 1, 'ce'], [30, 1, 'ce'], [0, 3, 'ce'], [16, 1, 'ue'], [21, 3, 'ue'], [33, 3, 'ce'], [30, 3, 'ue'], [29, 9, 'ue'], [1, 9, 'ce']], 15, 2], [[1, 16], [3, 21], [9, 29]]], ['control [[[38, 2, "ce"], [26, 10, "ce"], [5, 1, "ce"], [32, 2, "ce"]], 5, 3]', [[[38, 2, 'ce'], [26, 10, 'ce'], [5, 1, 'ce'], [32, 2, 'ce']], 5, 3], []], ['control [[[1, 10, "ce"], [11, 10, "ue"], [29, 10, "ce"]], 5, 2]', [[[1, 10, 'ce'], [11, 10, 'ue'], [29, 10, 'ce']], 5, 2], [[10, 11]]], ['control [[[3, 3, "ue"], [24, 2, "ce"], [16, 1, "ue"], [12, 3, "ce"], [18, 3...', [[[3, 3, 'ue'], [24, 2, 'ce'], [16, 1, 'ue'], [12, 3, 'ce'], [18, 3, 'ce'], [3, 1, 'ce'], [2, 3, 'ce']], 5, 3], [[1, 16], [3, 3]]], ['control [[[32, 2, "ce"], [16, 10, "ue"], [26, 1, "ue"], [3, 10, "ce"], [1, ...', [[[32, 2, 'ce'], [16, 10, 'ue'], [26, 1, 'ue'], [3, 10, 'ce'], [1, 2, 'ce']], 10, 3], [[1, 26], [10, 16]]], ['control [[[15, 3, "ce"], [35, 9, "ce"], [0, 3, "ue"]], 5, 2]', [[[15, 3, 'ce'], [35, 9, 'ce'], [0, 3, 'ue']], 5, 2], [[3, 0]]]], [['regression [[[32, 2, "ue"], [1, 1, "ue"], [28, 9, "ce"], [20, 10, "ce"], [13, ...', [[[32, 2, 'ue'], [1, 1, 'ue'], [28, 9, 'ce'], [20, 10, 'ce'], [13, 10, 'ce'], [28, 10, 'ce'], [6, 9, 'ce'], [36, 3, 'ce']], 15, 2], [[1, 1], [2, 32], [10, 20]]], ['regression [[[5, 2, "ue"], [6, 2, "ce"], [7, 2, "ue"]], 10, 3]', [[[5, 2, 'ue'], [6, 2, 'ce'], [7, 2, 'ue']], 10, 3], [[2, 5]]], ['partial-repair [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...', [[[21, 9, 'ce'], [27, 2, 'ue'], [0, 2, 'ue'], [21, 1, 'ce'], [9, 10, 'ce'], [0, 1, 'ce']], 5, 2], [[2, 0]]], ['partial-repair [[[30, 1, "ce"], [30, 1, "ce"], [0, 3, "ce"], [16, 1, "ue"], [21, 3...', [[[30, 1, 'ce'], [30, 1, 'ce'], [0, 3, 'ce'], [16, 1, 'ue'], [21, 3, 'ue'], [33, 3, 'ce'], [30, 3, 'ue'], [29, 9, 'ue'], [1, 9, 'ce']], 15, 2], [[1, 16], [3, 21], [9, 29]]], ['control [[[27, 3, "ce"], [2, 1, "ce"], [8, 1, "ce"], [28, 10, "ce"], [25, 1...', [[[27, 3, 'ce'], [2, 1, 'ce'], [8, 1, 'ce'], [28, 10, 'ce'], [25, 1, 'ce'], [19, 10, 'ce'], [8, 2, 'ue'], [7, 10, 'ce']], 5, 2], [[2, 8]]], ['control [[[3, 2, "ue"], [11, 1, "ce"], [29, 1, "ce"], [28, 3, "ce"]], 5, 2]', [[[3, 2, 'ue'], [11, 1, 'ce'], [29, 1, 'ce'], [28, 3, 'ce']], 5, 2], [[2, 3]]], ['control [[[2, 3, "ce"], [35, 9, "ce"], [27, 2, "ce"], [40, 10, "ce"], [38, ...', [[[2, 3, 'ce'], [35, 9, 'ce'], [27, 2, 'ce'], [40, 10, 'ce'], [38, 3, 'ce'], [23, 3, 'ue'], [7, 10, 'ce'], [29, 9, 'ue']], 15, 3], [[3, 23], [9, 29]]], ['control [[[12, 3, "ce"], [37, 2, "ce"], [33, 1, "ce"], [12, 10, "ce"], [4, ...', [[[12, 3, 'ce'], [37, 2, 'ce'], [33, 1, 'ce'], [12, 10, 'ce'], [4, 3, 'ce'], [7, 10, 'ce'], [34, 1, 'ue'], [1, 1, 'ce'], [11, 3, 'ce']], 5, 3], [[1, 34]]]], [['regression [[[30, 1, "ce"], [30, 1, "ce"], [0, 3, "ce"], [16, 1, "ue"], [21, 3...', [[[30, 1, 'ce'], [30, 1, 'ce'], [0, 3, 'ce'], [16, 1, 'ue'], [21, 3, 'ue'], [33, 3, 'ce'], [30, 3, 'ue'], [29, 9, 'ue'], [1, 9, 'ce']], 15, 2], [[1, 16], [3, 21], [9, 29]]], ['regression [[[32, 2, "ue"], [1, 1, "ue"], [28, 9, "ce"], [20, 10, "ce"], [13, ...', [[[32, 2, 'ue'], [1, 1, 'ue'], [28, 9, 'ce'], [20, 10, 'ce'], [13, 10, 'ce'], [28, 10, 'ce'], [6, 9, 'ce'], [36, 3, 'ce']], 15, 2], [[1, 1], [2, 32], [10, 20]]], ['partial-repair [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...', [[[21, 9, 'ce'], [27, 2, 'ue'], [0, 2, 'ue'], [21, 1, 'ce'], [9, 10, 'ce'], [0, 1, 'ce']], 5, 2], [[2, 0]]], ['control [[[26, 9, "ce"], [40, 3, "ce"], [26, 10, "ce"], [40, 9, "ce"], [23,...', [[[26, 9, 'ce'], [40, 3, 'ce'], [26, 10, 'ce'], [40, 9, 'ce'], [23, 10, 'ue'], [9, 10, 'ce'], [21, 9, 'ce'], [0, 3, 'ce'], [29, 9, 'ue'], [8, 1, 'ce']], 5, 3], [[9, 29], [10, 23]]], ['control [[[39, 10, "ce"], [4, 3, "ce"], [20, 2, "ce"], [33, 1, "ce"]], 15, 3]', [[[39, 10, 'ce'], [4, 3, 'ce'], [20, 2, 'ce'], [33, 1, 'ce']], 15, 3], []], ['control [[[0, 1, "ce"], [10, 1, "ce"], [20, 1, "ce"]], 10, 2]', [[[0, 1, 'ce'], [10, 1, 'ce'], [20, 1, 'ce']], 10, 2], []], ['control [[[0, 1, "ce"], [9, 1, "ce"]], 10, 2]', [[[0, 1, 'ce'], [9, 1, 'ce']], 10, 2], [[1, 9]]], ['control [[[3, 1, "ce"], [1, 1, "ce"], [2, 4, "ce"]], 5, 2]', [[[3, 1, 'ce'], [1, 1, 'ce'], [2, 4, 'ce']], 5, 2], [[1, 3]]]], [['regression [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...', [[[21, 9, 'ce'], [27, 2, 'ue'], [0, 2, 'ue'], [21, 1, 'ce'], [9, 10, 'ce'], [0, 1, 'ce']], 5, 2], [[2, 0]]], ['regression [[[30, 1, "ce"], [30, 1, "ce"], [0, 3, "ce"], [16, 1, "ue"], [21, 3...', [[[30, 1, 'ce'], [30, 1, 'ce'], [0, 3, 'ce'], [16, 1, 'ue'], [21, 3, 'ue'], [33, 3, 'ce'], [30, 3, 'ue'], [29, 9, 'ue'], [1, 9, 'ce']], 15, 2], [[1, 16], [3, 21], [9, 29]]], ['control [[[0, 1, "ce"], [1, 2, "ce"], [2, 3, "ce"]], 10, 2]', [[[0, 1, 'ce'], [1, 2, 'ce'], [2, 3, 'ce']], 10, 2], []], ['control [[[0, 8, "ce"], [1, 9, "ce"]], 10, 2]', [[[0, 8, 'ce'], [1, 9, 'ce']], 10, 2], []], ['control [[], 10, 2]', [[], 10, 2], []], ['control [[[39, 3, "ce"], [20, 10, "ce"], [16, 1, "ce"], [20, 10, "ce"], [22...', [[[39, 3, 'ce'], [20, 10, 'ce'], [16, 1, 'ce'], [20, 10, 'ce'], [22, 3, 'ue'], [40, 10, 'ce'], [15, 10, 'ce']], 5, 3], [[3, 22]]], ['control [[[14, 10, "ce"], [33, 1, "ce"], [3, 9, "ce"], [11, 1, "ce"], [9, 2...', [[[14, 10, 'ce'], [33, 1, 'ce'], [3, 9, 'ce'], [11, 1, 'ce'], [9, 2, 'ce'], [7, 9, 'ce'], [29, 3, 'ce'], [27, 1, 'ce']], 10, 3], []], ['control [[[35, 3, "ce"], [11, 1, "ce"], [8, 1, "ce"], [28, 3, "ce"], [3, 10...', [[[35, 3, 'ce'], [11, 1, 'ce'], [8, 1, 'ce'], [28, 3, 'ce'], [3, 10, 'ue'], [23, 3, 'ce'], [13, 10, 'ce']], 15, 3], [[3, 35], [10, 3]]]]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
regression [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...[[2, 27]][[2, 0]]Failed
regression [[[30, 1, "ce"], [30, 1, "ce"], [0, 3, "ce"], [16, 1, "ue"], [21, 3...[[1, 30], [3, 30], [9, 29]][[1, 16], [3, 21], [9, 29]]Failed
control [[[39, 3, "ce"], [20, 10, "ce"], [16, 1, "ce"], [20, 10, "ce"], [22...[[3, 22]][[3, 22]]Passed
control [[[14, 10, "ce"], [33, 1, "ce"], [3, 9, "ce"], [11, 1, "ce"], [9, 2...[][]Passed
control [[[35, 3, "ce"], [11, 1, "ce"], [8, 1, "ce"], [28, 3, "ce"], [3, 10...[[3, 35], [10, 3]][[3, 35], [10, 3]]Passed
control [[[2, 10, "ue"], [35, 1, "ce"], [6, 3, "ue"], [37, 10, "ce"]], 10, 3][[3, 6], [10, 2]][[3, 6], [10, 2]]Passed
control [[[5, 9, "ce"], [3, 10, "ue"], [23, 1, "ce"]], 15, 3][[10, 3]][[10, 3]]Passed
control [[[38, 2, "ce"], [26, 10, "ce"], [5, 1, "ce"], [32, 2, "ce"]], 5, 3][][]Passed

SHA-256 / f4454f14e95ce69b5a45ac26d254bcf1d4067809e681a61bb7118bc381cda35d

2 / The unsuccessful fix

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(events, window, threshold):
    retired = {}
    hist = {}
    for t, row, kind in sorted(events, key=lambda e: e[0]):
        if row in retired and kind == 'ce':
            continue
        if kind == 'ue':
            retired[row] = t
            continue
        h = [x for x in hist.get(row, []) if x > t - window] + [t]
        hist[row] = h
        if len(h) >= threshold:
            retired[row] = t
    return sorted([rw, tm] for rw, tm in retired.items())
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[['regression [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...', [[[21, 9, 'ce'], [27, 2, 'ue'], [0, 2, 'ue'], [21, 1, 'ce'], [9, 10, 'ce'], [0, 1, 'ce']], 5, 2], [[2, 0]]], ['regression [[[30, 1, "ce"], [30, 1, "ce"], [0, 3, "ce"], [16, 1, "ue"], [21, 3...', [[[30, 1, 'ce'], [30, 1, 'ce'], [0, 3, 'ce'], [16, 1, 'ue'], [21, 3, 'ue'], [33, 3, 'ce'], [30, 3, 'ue'], [29, 9, 'ue'], [1, 9, 'ce']], 15, 2], [[1, 16], [3, 21], [9, 29]]], ['control [[[39, 3, "ce"], [20, 10, "ce"], [16, 1, "ce"], [20, 10, "ce"], [22...', [[[39, 3, 'ce'], [20, 10, 'ce'], [16, 1, 'ce'], [20, 10, 'ce'], [22, 3, 'ue'], [40, 10, 'ce'], [15, 10, 'ce']], 5, 3], [[3, 22]]], ['control [[[14, 10, "ce"], [33, 1, "ce"], [3, 9, "ce"], [11, 1, "ce"], [9, 2...', [[[14, 10, 'ce'], [33, 1, 'ce'], [3, 9, 'ce'], [11, 1, 'ce'], [9, 2, 'ce'], [7, 9, 'ce'], [29, 3, 'ce'], [27, 1, 'ce']], 10, 3], []], ['control [[[35, 3, "ce"], [11, 1, "ce"], [8, 1, "ce"], [28, 3, "ce"], [3, 10...', [[[35, 3, 'ce'], [11, 1, 'ce'], [8, 1, 'ce'], [28, 3, 'ce'], [3, 10, 'ue'], [23, 3, 'ce'], [13, 10, 'ce']], 15, 3], [[3, 35], [10, 3]]], ['control [[[2, 10, "ue"], [35, 1, "ce"], [6, 3, "ue"], [37, 10, "ce"]], 10, 3]', [[[2, 10, 'ue'], [35, 1, 'ce'], [6, 3, 'ue'], [37, 10, 'ce']], 10, 3], [[3, 6], [10, 2]]], ['control [[[5, 9, "ce"], [3, 10, "ue"], [23, 1, "ce"]], 15, 3]', [[[5, 9, 'ce'], [3, 10, 'ue'], [23, 1, 'ce']], 15, 3], [[10, 3]]], ['control [[[38, 2, "ce"], [26, 10, "ce"], [5, 1, "ce"], [32, 2, "ce"]], 5, 3]', [[[38, 2, 'ce'], [26, 10, 'ce'], [5, 1, 'ce'], [32, 2, 'ce']], 5, 3], []]], [['regression [[[5, 2, "ue"], [6, 2, "ce"], [7, 2, "ue"]], 10, 3]', [[[5, 2, 'ue'], [6, 2, 'ce'], [7, 2, 'ue']], 10, 3], [[2, 5]]], ['regression [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...', [[[21, 9, 'ce'], [27, 2, 'ue'], [0, 2, 'ue'], [21, 1, 'ce'], [9, 10, 'ce'], [0, 1, 'ce']], 5, 2], [[2, 0]]], ['partial-repair [[[30, 1, "ce"], [30, 1, "ce"], [0, 3, "ce"], [16, 1, "ue"], [21, 3...', [[[30, 1, 'ce'], [30, 1, 'ce'], [0, 3, 'ce'], [16, 1, 'ue'], [21, 3, 'ue'], [33, 3, 'ce'], [30, 3, 'ue'], [29, 9, 'ue'], [1, 9, 'ce']], 15, 2], [[1, 16], [3, 21], [9, 29]]], ['control [[[38, 2, "ce"], [26, 10, "ce"], [5, 1, "ce"], [32, 2, "ce"]], 5, 3]', [[[38, 2, 'ce'], [26, 10, 'ce'], [5, 1, 'ce'], [32, 2, 'ce']], 5, 3], []], ['control [[[1, 10, "ce"], [11, 10, "ue"], [29, 10, "ce"]], 5, 2]', [[[1, 10, 'ce'], [11, 10, 'ue'], [29, 10, 'ce']], 5, 2], [[10, 11]]], ['control [[[3, 3, "ue"], [24, 2, "ce"], [16, 1, "ue"], [12, 3, "ce"], [18, 3...', [[[3, 3, 'ue'], [24, 2, 'ce'], [16, 1, 'ue'], [12, 3, 'ce'], [18, 3, 'ce'], [3, 1, 'ce'], [2, 3, 'ce']], 5, 3], [[1, 16], [3, 3]]], ['control [[[32, 2, "ce"], [16, 10, "ue"], [26, 1, "ue"], [3, 10, "ce"], [1, ...', [[[32, 2, 'ce'], [16, 10, 'ue'], [26, 1, 'ue'], [3, 10, 'ce'], [1, 2, 'ce']], 10, 3], [[1, 26], [10, 16]]], ['control [[[15, 3, "ce"], [35, 9, "ce"], [0, 3, "ue"]], 5, 2]', [[[15, 3, 'ce'], [35, 9, 'ce'], [0, 3, 'ue']], 5, 2], [[3, 0]]]], [['regression [[[32, 2, "ue"], [1, 1, "ue"], [28, 9, "ce"], [20, 10, "ce"], [13, ...', [[[32, 2, 'ue'], [1, 1, 'ue'], [28, 9, 'ce'], [20, 10, 'ce'], [13, 10, 'ce'], [28, 10, 'ce'], [6, 9, 'ce'], [36, 3, 'ce']], 15, 2], [[1, 1], [2, 32], [10, 20]]], ['regression [[[5, 2, "ue"], [6, 2, "ce"], [7, 2, "ue"]], 10, 3]', [[[5, 2, 'ue'], [6, 2, 'ce'], [7, 2, 'ue']], 10, 3], [[2, 5]]], ['partial-repair [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...', [[[21, 9, 'ce'], [27, 2, 'ue'], [0, 2, 'ue'], [21, 1, 'ce'], [9, 10, 'ce'], [0, 1, 'ce']], 5, 2], [[2, 0]]], ['partial-repair [[[30, 1, "ce"], [30, 1, "ce"], [0, 3, "ce"], [16, 1, "ue"], [21, 3...', [[[30, 1, 'ce'], [30, 1, 'ce'], [0, 3, 'ce'], [16, 1, 'ue'], [21, 3, 'ue'], [33, 3, 'ce'], [30, 3, 'ue'], [29, 9, 'ue'], [1, 9, 'ce']], 15, 2], [[1, 16], [3, 21], [9, 29]]], ['control [[[27, 3, "ce"], [2, 1, "ce"], [8, 1, "ce"], [28, 10, "ce"], [25, 1...', [[[27, 3, 'ce'], [2, 1, 'ce'], [8, 1, 'ce'], [28, 10, 'ce'], [25, 1, 'ce'], [19, 10, 'ce'], [8, 2, 'ue'], [7, 10, 'ce']], 5, 2], [[2, 8]]], ['control [[[3, 2, "ue"], [11, 1, "ce"], [29, 1, "ce"], [28, 3, "ce"]], 5, 2]', [[[3, 2, 'ue'], [11, 1, 'ce'], [29, 1, 'ce'], [28, 3, 'ce']], 5, 2], [[2, 3]]], ['control [[[2, 3, "ce"], [35, 9, "ce"], [27, 2, "ce"], [40, 10, "ce"], [38, ...', [[[2, 3, 'ce'], [35, 9, 'ce'], [27, 2, 'ce'], [40, 10, 'ce'], [38, 3, 'ce'], [23, 3, 'ue'], [7, 10, 'ce'], [29, 9, 'ue']], 15, 3], [[3, 23], [9, 29]]], ['control [[[12, 3, "ce"], [37, 2, "ce"], [33, 1, "ce"], [12, 10, "ce"], [4, ...', [[[12, 3, 'ce'], [37, 2, 'ce'], [33, 1, 'ce'], [12, 10, 'ce'], [4, 3, 'ce'], [7, 10, 'ce'], [34, 1, 'ue'], [1, 1, 'ce'], [11, 3, 'ce']], 5, 3], [[1, 34]]]], [['regression [[[30, 1, "ce"], [30, 1, "ce"], [0, 3, "ce"], [16, 1, "ue"], [21, 3...', [[[30, 1, 'ce'], [30, 1, 'ce'], [0, 3, 'ce'], [16, 1, 'ue'], [21, 3, 'ue'], [33, 3, 'ce'], [30, 3, 'ue'], [29, 9, 'ue'], [1, 9, 'ce']], 15, 2], [[1, 16], [3, 21], [9, 29]]], ['regression [[[32, 2, "ue"], [1, 1, "ue"], [28, 9, "ce"], [20, 10, "ce"], [13, ...', [[[32, 2, 'ue'], [1, 1, 'ue'], [28, 9, 'ce'], [20, 10, 'ce'], [13, 10, 'ce'], [28, 10, 'ce'], [6, 9, 'ce'], [36, 3, 'ce']], 15, 2], [[1, 1], [2, 32], [10, 20]]], ['partial-repair [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...', [[[21, 9, 'ce'], [27, 2, 'ue'], [0, 2, 'ue'], [21, 1, 'ce'], [9, 10, 'ce'], [0, 1, 'ce']], 5, 2], [[2, 0]]], ['control [[[26, 9, "ce"], [40, 3, "ce"], [26, 10, "ce"], [40, 9, "ce"], [23,...', [[[26, 9, 'ce'], [40, 3, 'ce'], [26, 10, 'ce'], [40, 9, 'ce'], [23, 10, 'ue'], [9, 10, 'ce'], [21, 9, 'ce'], [0, 3, 'ce'], [29, 9, 'ue'], [8, 1, 'ce']], 5, 3], [[9, 29], [10, 23]]], ['control [[[39, 10, "ce"], [4, 3, "ce"], [20, 2, "ce"], [33, 1, "ce"]], 15, 3]', [[[39, 10, 'ce'], [4, 3, 'ce'], [20, 2, 'ce'], [33, 1, 'ce']], 15, 3], []], ['control [[[0, 1, "ce"], [10, 1, "ce"], [20, 1, "ce"]], 10, 2]', [[[0, 1, 'ce'], [10, 1, 'ce'], [20, 1, 'ce']], 10, 2], []], ['control [[[0, 1, "ce"], [9, 1, "ce"]], 10, 2]', [[[0, 1, 'ce'], [9, 1, 'ce']], 10, 2], [[1, 9]]], ['control [[[3, 1, "ce"], [1, 1, "ce"], [2, 4, "ce"]], 5, 2]', [[[3, 1, 'ce'], [1, 1, 'ce'], [2, 4, 'ce']], 5, 2], [[1, 3]]]], [['regression [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...', [[[21, 9, 'ce'], [27, 2, 'ue'], [0, 2, 'ue'], [21, 1, 'ce'], [9, 10, 'ce'], [0, 1, 'ce']], 5, 2], [[2, 0]]], ['regression [[[30, 1, "ce"], [30, 1, "ce"], [0, 3, "ce"], [16, 1, "ue"], [21, 3...', [[[30, 1, 'ce'], [30, 1, 'ce'], [0, 3, 'ce'], [16, 1, 'ue'], [21, 3, 'ue'], [33, 3, 'ce'], [30, 3, 'ue'], [29, 9, 'ue'], [1, 9, 'ce']], 15, 2], [[1, 16], [3, 21], [9, 29]]], ['control [[[0, 1, "ce"], [1, 2, "ce"], [2, 3, "ce"]], 10, 2]', [[[0, 1, 'ce'], [1, 2, 'ce'], [2, 3, 'ce']], 10, 2], []], ['control [[[0, 8, "ce"], [1, 9, "ce"]], 10, 2]', [[[0, 8, 'ce'], [1, 9, 'ce']], 10, 2], []], ['control [[], 10, 2]', [[], 10, 2], []], ['control [[[39, 3, "ce"], [20, 10, "ce"], [16, 1, "ce"], [20, 10, "ce"], [22...', [[[39, 3, 'ce'], [20, 10, 'ce'], [16, 1, 'ce'], [20, 10, 'ce'], [22, 3, 'ue'], [40, 10, 'ce'], [15, 10, 'ce']], 5, 3], [[3, 22]]], ['control [[[14, 10, "ce"], [33, 1, "ce"], [3, 9, "ce"], [11, 1, "ce"], [9, 2...', [[[14, 10, 'ce'], [33, 1, 'ce'], [3, 9, 'ce'], [11, 1, 'ce'], [9, 2, 'ce'], [7, 9, 'ce'], [29, 3, 'ce'], [27, 1, 'ce']], 10, 3], []], ['control [[[35, 3, "ce"], [11, 1, "ce"], [8, 1, "ce"], [28, 3, "ce"], [3, 10...', [[[35, 3, 'ce'], [11, 1, 'ce'], [8, 1, 'ce'], [28, 3, 'ce'], [3, 10, 'ue'], [23, 3, 'ce'], [13, 10, 'ce']], 15, 3], [[3, 35], [10, 3]]]]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
regression [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...[[2, 27]][[2, 0]]Failed
regression [[[30, 1, "ce"], [30, 1, "ce"], [0, 3, "ce"], [16, 1, "ue"], [21, 3...[[1, 16], [3, 30], [9, 29]][[1, 16], [3, 21], [9, 29]]Failed
control [[[39, 3, "ce"], [20, 10, "ce"], [16, 1, "ce"], [20, 10, "ce"], [22...[[3, 22]][[3, 22]]Passed
control [[[14, 10, "ce"], [33, 1, "ce"], [3, 9, "ce"], [11, 1, "ce"], [9, 2...[][]Passed
control [[[35, 3, "ce"], [11, 1, "ce"], [8, 1, "ce"], [28, 3, "ce"], [3, 10...[[3, 35], [10, 3]][[3, 35], [10, 3]]Passed
control [[[2, 10, "ue"], [35, 1, "ce"], [6, 3, "ue"], [37, 10, "ce"]], 10, 3][[3, 6], [10, 2]][[3, 6], [10, 2]]Passed
control [[[5, 9, "ce"], [3, 10, "ue"], [23, 1, "ce"]], 15, 3][[10, 3]][[10, 3]]Passed
control [[[38, 2, "ce"], [26, 10, "ce"], [5, 1, "ce"], [32, 2, "ce"]], 5, 3][][]Passed

SHA-256 / f9fdf8636e18ebe00287cfa39788491a44b31b97159c2def2ee216006ea834b5

3 / The verified repair

Exit 0
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(events, window, threshold):
    retired = {}
    hist = {}
    for t, row, kind in sorted(events, key=lambda e: e[0]):
        if row in retired:
            continue
        if kind == 'ue':
            retired[row] = t
            continue
        h = [x for x in hist.get(row, []) if x > t - window] + [t]
        hist[row] = h
        if len(h) >= threshold:
            retired[row] = t
    return sorted([rw, tm] for rw, tm in retired.items())
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[['regression [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...', [[[21, 9, 'ce'], [27, 2, 'ue'], [0, 2, 'ue'], [21, 1, 'ce'], [9, 10, 'ce'], [0, 1, 'ce']], 5, 2], [[2, 0]]], ['regression [[[30, 1, "ce"], [30, 1, "ce"], [0, 3, "ce"], [16, 1, "ue"], [21, 3...', [[[30, 1, 'ce'], [30, 1, 'ce'], [0, 3, 'ce'], [16, 1, 'ue'], [21, 3, 'ue'], [33, 3, 'ce'], [30, 3, 'ue'], [29, 9, 'ue'], [1, 9, 'ce']], 15, 2], [[1, 16], [3, 21], [9, 29]]], ['control [[[39, 3, "ce"], [20, 10, "ce"], [16, 1, "ce"], [20, 10, "ce"], [22...', [[[39, 3, 'ce'], [20, 10, 'ce'], [16, 1, 'ce'], [20, 10, 'ce'], [22, 3, 'ue'], [40, 10, 'ce'], [15, 10, 'ce']], 5, 3], [[3, 22]]], ['control [[[14, 10, "ce"], [33, 1, "ce"], [3, 9, "ce"], [11, 1, "ce"], [9, 2...', [[[14, 10, 'ce'], [33, 1, 'ce'], [3, 9, 'ce'], [11, 1, 'ce'], [9, 2, 'ce'], [7, 9, 'ce'], [29, 3, 'ce'], [27, 1, 'ce']], 10, 3], []], ['control [[[35, 3, "ce"], [11, 1, "ce"], [8, 1, "ce"], [28, 3, "ce"], [3, 10...', [[[35, 3, 'ce'], [11, 1, 'ce'], [8, 1, 'ce'], [28, 3, 'ce'], [3, 10, 'ue'], [23, 3, 'ce'], [13, 10, 'ce']], 15, 3], [[3, 35], [10, 3]]], ['control [[[2, 10, "ue"], [35, 1, "ce"], [6, 3, "ue"], [37, 10, "ce"]], 10, 3]', [[[2, 10, 'ue'], [35, 1, 'ce'], [6, 3, 'ue'], [37, 10, 'ce']], 10, 3], [[3, 6], [10, 2]]], ['control [[[5, 9, "ce"], [3, 10, "ue"], [23, 1, "ce"]], 15, 3]', [[[5, 9, 'ce'], [3, 10, 'ue'], [23, 1, 'ce']], 15, 3], [[10, 3]]], ['control [[[38, 2, "ce"], [26, 10, "ce"], [5, 1, "ce"], [32, 2, "ce"]], 5, 3]', [[[38, 2, 'ce'], [26, 10, 'ce'], [5, 1, 'ce'], [32, 2, 'ce']], 5, 3], []]], [['regression [[[5, 2, "ue"], [6, 2, "ce"], [7, 2, "ue"]], 10, 3]', [[[5, 2, 'ue'], [6, 2, 'ce'], [7, 2, 'ue']], 10, 3], [[2, 5]]], ['regression [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...', [[[21, 9, 'ce'], [27, 2, 'ue'], [0, 2, 'ue'], [21, 1, 'ce'], [9, 10, 'ce'], [0, 1, 'ce']], 5, 2], [[2, 0]]], ['partial-repair [[[30, 1, "ce"], [30, 1, "ce"], [0, 3, "ce"], [16, 1, "ue"], [21, 3...', [[[30, 1, 'ce'], [30, 1, 'ce'], [0, 3, 'ce'], [16, 1, 'ue'], [21, 3, 'ue'], [33, 3, 'ce'], [30, 3, 'ue'], [29, 9, 'ue'], [1, 9, 'ce']], 15, 2], [[1, 16], [3, 21], [9, 29]]], ['control [[[38, 2, "ce"], [26, 10, "ce"], [5, 1, "ce"], [32, 2, "ce"]], 5, 3]', [[[38, 2, 'ce'], [26, 10, 'ce'], [5, 1, 'ce'], [32, 2, 'ce']], 5, 3], []], ['control [[[1, 10, "ce"], [11, 10, "ue"], [29, 10, "ce"]], 5, 2]', [[[1, 10, 'ce'], [11, 10, 'ue'], [29, 10, 'ce']], 5, 2], [[10, 11]]], ['control [[[3, 3, "ue"], [24, 2, "ce"], [16, 1, "ue"], [12, 3, "ce"], [18, 3...', [[[3, 3, 'ue'], [24, 2, 'ce'], [16, 1, 'ue'], [12, 3, 'ce'], [18, 3, 'ce'], [3, 1, 'ce'], [2, 3, 'ce']], 5, 3], [[1, 16], [3, 3]]], ['control [[[32, 2, "ce"], [16, 10, "ue"], [26, 1, "ue"], [3, 10, "ce"], [1, ...', [[[32, 2, 'ce'], [16, 10, 'ue'], [26, 1, 'ue'], [3, 10, 'ce'], [1, 2, 'ce']], 10, 3], [[1, 26], [10, 16]]], ['control [[[15, 3, "ce"], [35, 9, "ce"], [0, 3, "ue"]], 5, 2]', [[[15, 3, 'ce'], [35, 9, 'ce'], [0, 3, 'ue']], 5, 2], [[3, 0]]]], [['regression [[[32, 2, "ue"], [1, 1, "ue"], [28, 9, "ce"], [20, 10, "ce"], [13, ...', [[[32, 2, 'ue'], [1, 1, 'ue'], [28, 9, 'ce'], [20, 10, 'ce'], [13, 10, 'ce'], [28, 10, 'ce'], [6, 9, 'ce'], [36, 3, 'ce']], 15, 2], [[1, 1], [2, 32], [10, 20]]], ['regression [[[5, 2, "ue"], [6, 2, "ce"], [7, 2, "ue"]], 10, 3]', [[[5, 2, 'ue'], [6, 2, 'ce'], [7, 2, 'ue']], 10, 3], [[2, 5]]], ['partial-repair [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...', [[[21, 9, 'ce'], [27, 2, 'ue'], [0, 2, 'ue'], [21, 1, 'ce'], [9, 10, 'ce'], [0, 1, 'ce']], 5, 2], [[2, 0]]], ['partial-repair [[[30, 1, "ce"], [30, 1, "ce"], [0, 3, "ce"], [16, 1, "ue"], [21, 3...', [[[30, 1, 'ce'], [30, 1, 'ce'], [0, 3, 'ce'], [16, 1, 'ue'], [21, 3, 'ue'], [33, 3, 'ce'], [30, 3, 'ue'], [29, 9, 'ue'], [1, 9, 'ce']], 15, 2], [[1, 16], [3, 21], [9, 29]]], ['control [[[27, 3, "ce"], [2, 1, "ce"], [8, 1, "ce"], [28, 10, "ce"], [25, 1...', [[[27, 3, 'ce'], [2, 1, 'ce'], [8, 1, 'ce'], [28, 10, 'ce'], [25, 1, 'ce'], [19, 10, 'ce'], [8, 2, 'ue'], [7, 10, 'ce']], 5, 2], [[2, 8]]], ['control [[[3, 2, "ue"], [11, 1, "ce"], [29, 1, "ce"], [28, 3, "ce"]], 5, 2]', [[[3, 2, 'ue'], [11, 1, 'ce'], [29, 1, 'ce'], [28, 3, 'ce']], 5, 2], [[2, 3]]], ['control [[[2, 3, "ce"], [35, 9, "ce"], [27, 2, "ce"], [40, 10, "ce"], [38, ...', [[[2, 3, 'ce'], [35, 9, 'ce'], [27, 2, 'ce'], [40, 10, 'ce'], [38, 3, 'ce'], [23, 3, 'ue'], [7, 10, 'ce'], [29, 9, 'ue']], 15, 3], [[3, 23], [9, 29]]], ['control [[[12, 3, "ce"], [37, 2, "ce"], [33, 1, "ce"], [12, 10, "ce"], [4, ...', [[[12, 3, 'ce'], [37, 2, 'ce'], [33, 1, 'ce'], [12, 10, 'ce'], [4, 3, 'ce'], [7, 10, 'ce'], [34, 1, 'ue'], [1, 1, 'ce'], [11, 3, 'ce']], 5, 3], [[1, 34]]]], [['regression [[[30, 1, "ce"], [30, 1, "ce"], [0, 3, "ce"], [16, 1, "ue"], [21, 3...', [[[30, 1, 'ce'], [30, 1, 'ce'], [0, 3, 'ce'], [16, 1, 'ue'], [21, 3, 'ue'], [33, 3, 'ce'], [30, 3, 'ue'], [29, 9, 'ue'], [1, 9, 'ce']], 15, 2], [[1, 16], [3, 21], [9, 29]]], ['regression [[[32, 2, "ue"], [1, 1, "ue"], [28, 9, "ce"], [20, 10, "ce"], [13, ...', [[[32, 2, 'ue'], [1, 1, 'ue'], [28, 9, 'ce'], [20, 10, 'ce'], [13, 10, 'ce'], [28, 10, 'ce'], [6, 9, 'ce'], [36, 3, 'ce']], 15, 2], [[1, 1], [2, 32], [10, 20]]], ['partial-repair [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...', [[[21, 9, 'ce'], [27, 2, 'ue'], [0, 2, 'ue'], [21, 1, 'ce'], [9, 10, 'ce'], [0, 1, 'ce']], 5, 2], [[2, 0]]], ['control [[[26, 9, "ce"], [40, 3, "ce"], [26, 10, "ce"], [40, 9, "ce"], [23,...', [[[26, 9, 'ce'], [40, 3, 'ce'], [26, 10, 'ce'], [40, 9, 'ce'], [23, 10, 'ue'], [9, 10, 'ce'], [21, 9, 'ce'], [0, 3, 'ce'], [29, 9, 'ue'], [8, 1, 'ce']], 5, 3], [[9, 29], [10, 23]]], ['control [[[39, 10, "ce"], [4, 3, "ce"], [20, 2, "ce"], [33, 1, "ce"]], 15, 3]', [[[39, 10, 'ce'], [4, 3, 'ce'], [20, 2, 'ce'], [33, 1, 'ce']], 15, 3], []], ['control [[[0, 1, "ce"], [10, 1, "ce"], [20, 1, "ce"]], 10, 2]', [[[0, 1, 'ce'], [10, 1, 'ce'], [20, 1, 'ce']], 10, 2], []], ['control [[[0, 1, "ce"], [9, 1, "ce"]], 10, 2]', [[[0, 1, 'ce'], [9, 1, 'ce']], 10, 2], [[1, 9]]], ['control [[[3, 1, "ce"], [1, 1, "ce"], [2, 4, "ce"]], 5, 2]', [[[3, 1, 'ce'], [1, 1, 'ce'], [2, 4, 'ce']], 5, 2], [[1, 3]]]], [['regression [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...', [[[21, 9, 'ce'], [27, 2, 'ue'], [0, 2, 'ue'], [21, 1, 'ce'], [9, 10, 'ce'], [0, 1, 'ce']], 5, 2], [[2, 0]]], ['regression [[[30, 1, "ce"], [30, 1, "ce"], [0, 3, "ce"], [16, 1, "ue"], [21, 3...', [[[30, 1, 'ce'], [30, 1, 'ce'], [0, 3, 'ce'], [16, 1, 'ue'], [21, 3, 'ue'], [33, 3, 'ce'], [30, 3, 'ue'], [29, 9, 'ue'], [1, 9, 'ce']], 15, 2], [[1, 16], [3, 21], [9, 29]]], ['control [[[0, 1, "ce"], [1, 2, "ce"], [2, 3, "ce"]], 10, 2]', [[[0, 1, 'ce'], [1, 2, 'ce'], [2, 3, 'ce']], 10, 2], []], ['control [[[0, 8, "ce"], [1, 9, "ce"]], 10, 2]', [[[0, 8, 'ce'], [1, 9, 'ce']], 10, 2], []], ['control [[], 10, 2]', [[], 10, 2], []], ['control [[[39, 3, "ce"], [20, 10, "ce"], [16, 1, "ce"], [20, 10, "ce"], [22...', [[[39, 3, 'ce'], [20, 10, 'ce'], [16, 1, 'ce'], [20, 10, 'ce'], [22, 3, 'ue'], [40, 10, 'ce'], [15, 10, 'ce']], 5, 3], [[3, 22]]], ['control [[[14, 10, "ce"], [33, 1, "ce"], [3, 9, "ce"], [11, 1, "ce"], [9, 2...', [[[14, 10, 'ce'], [33, 1, 'ce'], [3, 9, 'ce'], [11, 1, 'ce'], [9, 2, 'ce'], [7, 9, 'ce'], [29, 3, 'ce'], [27, 1, 'ce']], 10, 3], []], ['control [[[35, 3, "ce"], [11, 1, "ce"], [8, 1, "ce"], [28, 3, "ce"], [3, 10...', [[[35, 3, 'ce'], [11, 1, 'ce'], [8, 1, 'ce'], [28, 3, 'ce'], [3, 10, 'ue'], [23, 3, 'ce'], [13, 10, 'ce']], 15, 3], [[3, 35], [10, 3]]]]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
regression [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...[[2, 0]][[2, 0]]Passed
regression [[[30, 1, "ce"], [30, 1, "ce"], [0, 3, "ce"], [16, 1, "ue"], [21, 3...[[1, 16], [3, 21], [9, 29]][[1, 16], [3, 21], [9, 29]]Passed
control [[[39, 3, "ce"], [20, 10, "ce"], [16, 1, "ce"], [20, 10, "ce"], [22...[[3, 22]][[3, 22]]Passed
control [[[14, 10, "ce"], [33, 1, "ce"], [3, 9, "ce"], [11, 1, "ce"], [9, 2...[][]Passed
control [[[35, 3, "ce"], [11, 1, "ce"], [8, 1, "ce"], [28, 3, "ce"], [3, 10...[[3, 35], [10, 3]][[3, 35], [10, 3]]Passed
control [[[2, 10, "ue"], [35, 1, "ce"], [6, 3, "ue"], [37, 10, "ce"]], 10, 3][[3, 6], [10, 2]][[3, 6], [10, 2]]Passed
control [[[5, 9, "ce"], [3, 10, "ue"], [23, 1, "ce"]], 15, 3][[10, 3]][[10, 3]]Passed
control [[[38, 2, "ce"], [26, 10, "ce"], [5, 1, "ce"], [32, 2, "ce"]], 5, 3][][]Passed

SHA-256 / 81e4c5abef49558417a92882c99125296b4a1a3d96289a74fa429853b6c7a3ff

Verification & scope

A deterministic, bounded teaching model of the named code under the stated contract; not a production codec. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:36.292820+00:00.

Case digest / 74f9e8a9061553215716eacb6b525c82d95322f30d7aebbe383922adf6b3d8a5