FAILURE MAP
← Case archive

FA-72201 / Error-correcting codes / Open access

Row retirement processes events in arrival order · case 01

Out-of-order logs expire or count the wrong errors.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

Events are processed in list order instead of by timestamp.

VERIFIED REPAIR

Sort events by time (stably) before applying the policy.

Unsuccessful approach: Sorting by row groups events but loses their time order within the window.

Case contract

Memory-controller ECC policy. events are [time, row, kind] with kind "ce" (corrected) or "ue" (uncorrectable), processed in time order (stable for equal times). A "ue" retires its row immediately. A "ce" is added to the row's history, entries at or before time - window expire, and the row retires when the remaining count reaches the threshold. Events for a retired row are ignored. Return sorted [row, retirement time] pairs.

Why this case matters

Servers retire DRAM rows that show repeated corrected errors before they turn uncorrectable.

1 / The failure

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(events, window, threshold):
    retired = {}
    hist = {}
    for t, row, kind in events:
        if row in retired:
            continue
        if kind == 'ue':
            retired[row] = t
            continue
        h = [x for x in hist.get(row, []) if x > t - window] + [t]
        hist[row] = h
        if len(h) >= threshold:
            retired[row] = t
    return sorted([rw, tm] for rw, tm in retired.items())
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[['regression [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...', [[[21, 9, 'ce'], [27, 2, 'ue'], [0, 2, 'ue'], [21, 1, 'ce'], [9, 10, 'ce'], [0, 1, 'ce']], 5, 2], [[2, 0]]], ['regression [[[35, 3, "ce"], [11, 1, "ce"], [8, 1, "ce"], [28, 3, "ce"], [3, 10...', [[[35, 3, 'ce'], [11, 1, 'ce'], [8, 1, 'ce'], [28, 3, 'ce'], [3, 10, 'ue'], [23, 3, 'ce'], [13, 10, 'ce']], 15, 3], [[3, 35], [10, 3]]], ['control [[[39, 3, "ce"], [20, 10, "ce"], [16, 1, "ce"], [20, 10, "ce"], [22...', [[[39, 3, 'ce'], [20, 10, 'ce'], [16, 1, 'ce'], [20, 10, 'ce'], [22, 3, 'ue'], [40, 10, 'ce'], [15, 10, 'ce']], 5, 3], [[3, 22]]], ['control [[[14, 10, "ce"], [33, 1, "ce"], [3, 9, "ce"], [11, 1, "ce"], [9, 2...', [[[14, 10, 'ce'], [33, 1, 'ce'], [3, 9, 'ce'], [11, 1, 'ce'], [9, 2, 'ce'], [7, 9, 'ce'], [29, 3, 'ce'], [27, 1, 'ce']], 10, 3], []], ['control [[[2, 10, "ue"], [35, 1, "ce"], [6, 3, "ue"], [37, 10, "ce"]], 10, 3]', [[[2, 10, 'ue'], [35, 1, 'ce'], [6, 3, 'ue'], [37, 10, 'ce']], 10, 3], [[3, 6], [10, 2]]], ['control [[[5, 9, "ce"], [3, 10, "ue"], [23, 1, "ce"]], 15, 3]', [[[5, 9, 'ce'], [3, 10, 'ue'], [23, 1, 'ce']], 15, 3], [[10, 3]]], ['control [[[38, 2, "ce"], [26, 10, "ce"], [5, 1, "ce"], [32, 2, "ce"]], 5, 3]', [[[38, 2, 'ce'], [26, 10, 'ce'], [5, 1, 'ce'], [32, 2, 'ce']], 5, 3], []], ['control [[[1, 10, "ce"], [11, 10, "ue"], [29, 10, "ce"]], 5, 2]', [[[1, 10, 'ce'], [11, 10, 'ue'], [29, 10, 'ce']], 5, 2], [[10, 11]]]], [['regression [[[27, 3, "ce"], [2, 1, "ce"], [8, 1, "ce"], [28, 10, "ce"], [25, 1...', [[[27, 3, 'ce'], [2, 1, 'ce'], [8, 1, 'ce'], [28, 10, 'ce'], [25, 1, 'ce'], [19, 10, 'ce'], [8, 2, 'ue'], [7, 10, 'ce']], 5, 2], [[2, 8]]], ['regression [[[20, 1, "ce"], [37, 1, "ue"], [38, 10, "ce"], [24, 1, "ce"], [28,...', [[[20, 1, 'ce'], [37, 1, 'ue'], [38, 10, 'ce'], [24, 1, 'ce'], [28, 10, 'ce'], [7, 9, 'ce'], [7, 2, 'ce'], [22, 9, 'ce'], [31, 10, 'ce']], 10, 3], [[1, 37]]], ['control [[[1, 10, "ce"], [11, 10, "ue"], [29, 10, "ce"]], 5, 2]', [[[1, 10, 'ce'], [11, 10, 'ue'], [29, 10, 'ce']], 5, 2], [[10, 11]]], ['control [[[3, 3, "ue"], [24, 2, "ce"], [16, 1, "ue"], [12, 3, "ce"], [18, 3...', [[[3, 3, 'ue'], [24, 2, 'ce'], [16, 1, 'ue'], [12, 3, 'ce'], [18, 3, 'ce'], [3, 1, 'ce'], [2, 3, 'ce']], 5, 3], [[1, 16], [3, 3]]], ['control [[[32, 2, "ce"], [16, 10, "ue"], [26, 1, "ue"], [3, 10, "ce"], [1, ...', [[[32, 2, 'ce'], [16, 10, 'ue'], [26, 1, 'ue'], [3, 10, 'ce'], [1, 2, 'ce']], 10, 3], [[1, 26], [10, 16]]], ['control [[[15, 3, "ce"], [35, 9, "ce"], [0, 3, "ue"]], 5, 2]', [[[15, 3, 'ce'], [35, 9, 'ce'], [0, 3, 'ue']], 5, 2], [[3, 0]]], ['control [[[3, 2, "ue"], [11, 1, "ce"], [29, 1, "ce"], [28, 3, "ce"]], 5, 2]', [[[3, 2, 'ue'], [11, 1, 'ce'], [29, 1, 'ce'], [28, 3, 'ce']], 5, 2], [[2, 3]]], ['control [[[2, 3, "ce"], [35, 9, "ce"], [27, 2, "ce"], [40, 10, "ce"], [38, ...', [[[2, 3, 'ce'], [35, 9, 'ce'], [27, 2, 'ce'], [40, 10, 'ce'], [38, 3, 'ce'], [23, 3, 'ue'], [7, 10, 'ce'], [29, 9, 'ue']], 15, 3], [[3, 23], [9, 29]]]], [['regression [[[3, 1, "ce"], [1, 1, "ce"], [2, 4, "ce"]], 5, 2]', [[[3, 1, 'ce'], [1, 1, 'ce'], [2, 4, 'ce']], 5, 2], [[1, 3]]], ['regression [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...', [[[21, 9, 'ce'], [27, 2, 'ue'], [0, 2, 'ue'], [21, 1, 'ce'], [9, 10, 'ce'], [0, 1, 'ce']], 5, 2], [[2, 0]]], ['control [[[2, 3, "ce"], [35, 9, "ce"], [27, 2, "ce"], [40, 10, "ce"], [38, ...', [[[2, 3, 'ce'], [35, 9, 'ce'], [27, 2, 'ce'], [40, 10, 'ce'], [38, 3, 'ce'], [23, 3, 'ue'], [7, 10, 'ce'], [29, 9, 'ue']], 15, 3], [[3, 23], [9, 29]]], ['control [[[12, 3, "ce"], [37, 2, "ce"], [33, 1, "ce"], [12, 10, "ce"], [4, ...', [[[12, 3, 'ce'], [37, 2, 'ce'], [33, 1, 'ce'], [12, 10, 'ce'], [4, 3, 'ce'], [7, 10, 'ce'], [34, 1, 'ue'], [1, 1, 'ce'], [11, 3, 'ce']], 5, 3], [[1, 34]]], ['control [[[26, 9, "ce"], [40, 3, "ce"], [26, 10, "ce"], [40, 9, "ce"], [23,...', [[[26, 9, 'ce'], [40, 3, 'ce'], [26, 10, 'ce'], [40, 9, 'ce'], [23, 10, 'ue'], [9, 10, 'ce'], [21, 9, 'ce'], [0, 3, 'ce'], [29, 9, 'ue'], [8, 1, 'ce']], 5, 3], [[9, 29], [10, 23]]], ['control [[[39, 10, "ce"], [4, 3, "ce"], [20, 2, "ce"], [33, 1, "ce"]], 15, 3]', [[[39, 10, 'ce'], [4, 3, 'ce'], [20, 2, 'ce'], [33, 1, 'ce']], 15, 3], []], ['control [[[0, 1, "ce"], [10, 1, "ce"], [20, 1, "ce"]], 10, 2]', [[[0, 1, 'ce'], [10, 1, 'ce'], [20, 1, 'ce']], 10, 2], []], ['control [[[0, 1, "ce"], [9, 1, "ce"]], 10, 2]', [[[0, 1, 'ce'], [9, 1, 'ce']], 10, 2], [[1, 9]]]], [['regression [[[30, 1, "ce"], [30, 1, "ce"], [0, 3, "ce"], [16, 1, "ue"], [21, 3...', [[[30, 1, 'ce'], [30, 1, 'ce'], [0, 3, 'ce'], [16, 1, 'ue'], [21, 3, 'ue'], [33, 3, 'ce'], [30, 3, 'ue'], [29, 9, 'ue'], [1, 9, 'ce']], 15, 2], [[1, 16], [3, 21], [9, 29]]], ['regression [[[27, 3, "ce"], [2, 1, "ce"], [8, 1, "ce"], [28, 10, "ce"], [25, 1...', [[[27, 3, 'ce'], [2, 1, 'ce'], [8, 1, 'ce'], [28, 10, 'ce'], [25, 1, 'ce'], [19, 10, 'ce'], [8, 2, 'ue'], [7, 10, 'ce']], 5, 2], [[2, 8]]], ['control [[[0, 1, "ce"], [9, 1, "ce"]], 10, 2]', [[[0, 1, 'ce'], [9, 1, 'ce']], 10, 2], [[1, 9]]], ['control [[[5, 2, "ue"], [6, 2, "ce"], [7, 2, "ue"]], 10, 3]', [[[5, 2, 'ue'], [6, 2, 'ce'], [7, 2, 'ue']], 10, 3], [[2, 5]]], ['control [[[0, 1, "ce"], [1, 2, "ce"], [2, 3, "ce"]], 10, 2]', [[[0, 1, 'ce'], [1, 2, 'ce'], [2, 3, 'ce']], 10, 2], []], ['control [[[0, 8, "ce"], [1, 9, "ce"]], 10, 2]', [[[0, 8, 'ce'], [1, 9, 'ce']], 10, 2], []], ['control [[], 10, 2]', [[], 10, 2], []], ['control [[[39, 3, "ce"], [20, 10, "ce"], [16, 1, "ce"], [20, 10, "ce"], [22...', [[[39, 3, 'ce'], [20, 10, 'ce'], [16, 1, 'ce'], [20, 10, 'ce'], [22, 3, 'ue'], [40, 10, 'ce'], [15, 10, 'ce']], 5, 3], [[3, 22]]]], [['regression [[[32, 2, "ue"], [1, 1, "ue"], [28, 9, "ce"], [20, 10, "ce"], [13, ...', [[[32, 2, 'ue'], [1, 1, 'ue'], [28, 9, 'ce'], [20, 10, 'ce'], [13, 10, 'ce'], [28, 10, 'ce'], [6, 9, 'ce'], [36, 3, 'ce']], 15, 2], [[1, 1], [2, 32], [10, 20]]], ['regression [[[3, 1, "ce"], [1, 1, "ce"], [2, 4, "ce"]], 5, 2]', [[[3, 1, 'ce'], [1, 1, 'ce'], [2, 4, 'ce']], 5, 2], [[1, 3]]], ['control [[[39, 3, "ce"], [20, 10, "ce"], [16, 1, "ce"], [20, 10, "ce"], [22...', [[[39, 3, 'ce'], [20, 10, 'ce'], [16, 1, 'ce'], [20, 10, 'ce'], [22, 3, 'ue'], [40, 10, 'ce'], [15, 10, 'ce']], 5, 3], [[3, 22]]], ['control [[[14, 10, "ce"], [33, 1, "ce"], [3, 9, "ce"], [11, 1, "ce"], [9, 2...', [[[14, 10, 'ce'], [33, 1, 'ce'], [3, 9, 'ce'], [11, 1, 'ce'], [9, 2, 'ce'], [7, 9, 'ce'], [29, 3, 'ce'], [27, 1, 'ce']], 10, 3], []], ['control [[[2, 10, "ue"], [35, 1, "ce"], [6, 3, "ue"], [37, 10, "ce"]], 10, 3]', [[[2, 10, 'ue'], [35, 1, 'ce'], [6, 3, 'ue'], [37, 10, 'ce']], 10, 3], [[3, 6], [10, 2]]], ['control [[[5, 9, "ce"], [3, 10, "ue"], [23, 1, "ce"]], 15, 3]', [[[5, 9, 'ce'], [3, 10, 'ue'], [23, 1, 'ce']], 15, 3], [[10, 3]]], ['control [[[38, 2, "ce"], [26, 10, "ce"], [5, 1, "ce"], [32, 2, "ce"]], 5, 3]', [[[38, 2, 'ce'], [26, 10, 'ce'], [5, 1, 'ce'], [32, 2, 'ce']], 5, 3], []], ['control [[[1, 10, "ce"], [11, 10, "ue"], [29, 10, "ce"]], 5, 2]', [[[1, 10, 'ce'], [11, 10, 'ue'], [29, 10, 'ce']], 5, 2], [[10, 11]]]]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
regression [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...[[1, 0], [2, 27]][[2, 0]]Failed
regression [[[35, 3, "ce"], [11, 1, "ce"], [8, 1, "ce"], [28, 3, "ce"], [3, 10...[[3, 23], [10, 3]][[3, 35], [10, 3]]Failed
control [[[39, 3, "ce"], [20, 10, "ce"], [16, 1, "ce"], [20, 10, "ce"], [22...[[3, 22]][[3, 22]]Passed
control [[[14, 10, "ce"], [33, 1, "ce"], [3, 9, "ce"], [11, 1, "ce"], [9, 2...[][]Passed
control [[[2, 10, "ue"], [35, 1, "ce"], [6, 3, "ue"], [37, 10, "ce"]], 10, 3][[3, 6], [10, 2]][[3, 6], [10, 2]]Passed
control [[[5, 9, "ce"], [3, 10, "ue"], [23, 1, "ce"]], 15, 3][[10, 3]][[10, 3]]Passed
control [[[38, 2, "ce"], [26, 10, "ce"], [5, 1, "ce"], [32, 2, "ce"]], 5, 3][][]Passed
control [[[1, 10, "ce"], [11, 10, "ue"], [29, 10, "ce"]], 5, 2][[10, 11]][[10, 11]]Passed

SHA-256 / d67a1f3a2005fb4c5745b71e7d2192d51128d2044cfd901dfe23ed7467128de5

2 / The unsuccessful fix

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(events, window, threshold):
    retired = {}
    hist = {}
    for t, row, kind in sorted(events, key=lambda e: e[1]):
        if row in retired:
            continue
        if kind == 'ue':
            retired[row] = t
            continue
        h = [x for x in hist.get(row, []) if x > t - window] + [t]
        hist[row] = h
        if len(h) >= threshold:
            retired[row] = t
    return sorted([rw, tm] for rw, tm in retired.items())
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[['regression [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...', [[[21, 9, 'ce'], [27, 2, 'ue'], [0, 2, 'ue'], [21, 1, 'ce'], [9, 10, 'ce'], [0, 1, 'ce']], 5, 2], [[2, 0]]], ['regression [[[35, 3, "ce"], [11, 1, "ce"], [8, 1, "ce"], [28, 3, "ce"], [3, 10...', [[[35, 3, 'ce'], [11, 1, 'ce'], [8, 1, 'ce'], [28, 3, 'ce'], [3, 10, 'ue'], [23, 3, 'ce'], [13, 10, 'ce']], 15, 3], [[3, 35], [10, 3]]], ['control [[[39, 3, "ce"], [20, 10, "ce"], [16, 1, "ce"], [20, 10, "ce"], [22...', [[[39, 3, 'ce'], [20, 10, 'ce'], [16, 1, 'ce'], [20, 10, 'ce'], [22, 3, 'ue'], [40, 10, 'ce'], [15, 10, 'ce']], 5, 3], [[3, 22]]], ['control [[[14, 10, "ce"], [33, 1, "ce"], [3, 9, "ce"], [11, 1, "ce"], [9, 2...', [[[14, 10, 'ce'], [33, 1, 'ce'], [3, 9, 'ce'], [11, 1, 'ce'], [9, 2, 'ce'], [7, 9, 'ce'], [29, 3, 'ce'], [27, 1, 'ce']], 10, 3], []], ['control [[[2, 10, "ue"], [35, 1, "ce"], [6, 3, "ue"], [37, 10, "ce"]], 10, 3]', [[[2, 10, 'ue'], [35, 1, 'ce'], [6, 3, 'ue'], [37, 10, 'ce']], 10, 3], [[3, 6], [10, 2]]], ['control [[[5, 9, "ce"], [3, 10, "ue"], [23, 1, "ce"]], 15, 3]', [[[5, 9, 'ce'], [3, 10, 'ue'], [23, 1, 'ce']], 15, 3], [[10, 3]]], ['control [[[38, 2, "ce"], [26, 10, "ce"], [5, 1, "ce"], [32, 2, "ce"]], 5, 3]', [[[38, 2, 'ce'], [26, 10, 'ce'], [5, 1, 'ce'], [32, 2, 'ce']], 5, 3], []], ['control [[[1, 10, "ce"], [11, 10, "ue"], [29, 10, "ce"]], 5, 2]', [[[1, 10, 'ce'], [11, 10, 'ue'], [29, 10, 'ce']], 5, 2], [[10, 11]]]], [['regression [[[27, 3, "ce"], [2, 1, "ce"], [8, 1, "ce"], [28, 10, "ce"], [25, 1...', [[[27, 3, 'ce'], [2, 1, 'ce'], [8, 1, 'ce'], [28, 10, 'ce'], [25, 1, 'ce'], [19, 10, 'ce'], [8, 2, 'ue'], [7, 10, 'ce']], 5, 2], [[2, 8]]], ['regression [[[20, 1, "ce"], [37, 1, "ue"], [38, 10, "ce"], [24, 1, "ce"], [28,...', [[[20, 1, 'ce'], [37, 1, 'ue'], [38, 10, 'ce'], [24, 1, 'ce'], [28, 10, 'ce'], [7, 9, 'ce'], [7, 2, 'ce'], [22, 9, 'ce'], [31, 10, 'ce']], 10, 3], [[1, 37]]], ['control [[[1, 10, "ce"], [11, 10, "ue"], [29, 10, "ce"]], 5, 2]', [[[1, 10, 'ce'], [11, 10, 'ue'], [29, 10, 'ce']], 5, 2], [[10, 11]]], ['control [[[3, 3, "ue"], [24, 2, "ce"], [16, 1, "ue"], [12, 3, "ce"], [18, 3...', [[[3, 3, 'ue'], [24, 2, 'ce'], [16, 1, 'ue'], [12, 3, 'ce'], [18, 3, 'ce'], [3, 1, 'ce'], [2, 3, 'ce']], 5, 3], [[1, 16], [3, 3]]], ['control [[[32, 2, "ce"], [16, 10, "ue"], [26, 1, "ue"], [3, 10, "ce"], [1, ...', [[[32, 2, 'ce'], [16, 10, 'ue'], [26, 1, 'ue'], [3, 10, 'ce'], [1, 2, 'ce']], 10, 3], [[1, 26], [10, 16]]], ['control [[[15, 3, "ce"], [35, 9, "ce"], [0, 3, "ue"]], 5, 2]', [[[15, 3, 'ce'], [35, 9, 'ce'], [0, 3, 'ue']], 5, 2], [[3, 0]]], ['control [[[3, 2, "ue"], [11, 1, "ce"], [29, 1, "ce"], [28, 3, "ce"]], 5, 2]', [[[3, 2, 'ue'], [11, 1, 'ce'], [29, 1, 'ce'], [28, 3, 'ce']], 5, 2], [[2, 3]]], ['control [[[2, 3, "ce"], [35, 9, "ce"], [27, 2, "ce"], [40, 10, "ce"], [38, ...', [[[2, 3, 'ce'], [35, 9, 'ce'], [27, 2, 'ce'], [40, 10, 'ce'], [38, 3, 'ce'], [23, 3, 'ue'], [7, 10, 'ce'], [29, 9, 'ue']], 15, 3], [[3, 23], [9, 29]]]], [['regression [[[3, 1, "ce"], [1, 1, "ce"], [2, 4, "ce"]], 5, 2]', [[[3, 1, 'ce'], [1, 1, 'ce'], [2, 4, 'ce']], 5, 2], [[1, 3]]], ['regression [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...', [[[21, 9, 'ce'], [27, 2, 'ue'], [0, 2, 'ue'], [21, 1, 'ce'], [9, 10, 'ce'], [0, 1, 'ce']], 5, 2], [[2, 0]]], ['control [[[2, 3, "ce"], [35, 9, "ce"], [27, 2, "ce"], [40, 10, "ce"], [38, ...', [[[2, 3, 'ce'], [35, 9, 'ce'], [27, 2, 'ce'], [40, 10, 'ce'], [38, 3, 'ce'], [23, 3, 'ue'], [7, 10, 'ce'], [29, 9, 'ue']], 15, 3], [[3, 23], [9, 29]]], ['control [[[12, 3, "ce"], [37, 2, "ce"], [33, 1, "ce"], [12, 10, "ce"], [4, ...', [[[12, 3, 'ce'], [37, 2, 'ce'], [33, 1, 'ce'], [12, 10, 'ce'], [4, 3, 'ce'], [7, 10, 'ce'], [34, 1, 'ue'], [1, 1, 'ce'], [11, 3, 'ce']], 5, 3], [[1, 34]]], ['control [[[26, 9, "ce"], [40, 3, "ce"], [26, 10, "ce"], [40, 9, "ce"], [23,...', [[[26, 9, 'ce'], [40, 3, 'ce'], [26, 10, 'ce'], [40, 9, 'ce'], [23, 10, 'ue'], [9, 10, 'ce'], [21, 9, 'ce'], [0, 3, 'ce'], [29, 9, 'ue'], [8, 1, 'ce']], 5, 3], [[9, 29], [10, 23]]], ['control [[[39, 10, "ce"], [4, 3, "ce"], [20, 2, "ce"], [33, 1, "ce"]], 15, 3]', [[[39, 10, 'ce'], [4, 3, 'ce'], [20, 2, 'ce'], [33, 1, 'ce']], 15, 3], []], ['control [[[0, 1, "ce"], [10, 1, "ce"], [20, 1, "ce"]], 10, 2]', [[[0, 1, 'ce'], [10, 1, 'ce'], [20, 1, 'ce']], 10, 2], []], ['control [[[0, 1, "ce"], [9, 1, "ce"]], 10, 2]', [[[0, 1, 'ce'], [9, 1, 'ce']], 10, 2], [[1, 9]]]], [['regression [[[30, 1, "ce"], [30, 1, "ce"], [0, 3, "ce"], [16, 1, "ue"], [21, 3...', [[[30, 1, 'ce'], [30, 1, 'ce'], [0, 3, 'ce'], [16, 1, 'ue'], [21, 3, 'ue'], [33, 3, 'ce'], [30, 3, 'ue'], [29, 9, 'ue'], [1, 9, 'ce']], 15, 2], [[1, 16], [3, 21], [9, 29]]], ['regression [[[27, 3, "ce"], [2, 1, "ce"], [8, 1, "ce"], [28, 10, "ce"], [25, 1...', [[[27, 3, 'ce'], [2, 1, 'ce'], [8, 1, 'ce'], [28, 10, 'ce'], [25, 1, 'ce'], [19, 10, 'ce'], [8, 2, 'ue'], [7, 10, 'ce']], 5, 2], [[2, 8]]], ['control [[[0, 1, "ce"], [9, 1, "ce"]], 10, 2]', [[[0, 1, 'ce'], [9, 1, 'ce']], 10, 2], [[1, 9]]], ['control [[[5, 2, "ue"], [6, 2, "ce"], [7, 2, "ue"]], 10, 3]', [[[5, 2, 'ue'], [6, 2, 'ce'], [7, 2, 'ue']], 10, 3], [[2, 5]]], ['control [[[0, 1, "ce"], [1, 2, "ce"], [2, 3, "ce"]], 10, 2]', [[[0, 1, 'ce'], [1, 2, 'ce'], [2, 3, 'ce']], 10, 2], []], ['control [[[0, 8, "ce"], [1, 9, "ce"]], 10, 2]', [[[0, 8, 'ce'], [1, 9, 'ce']], 10, 2], []], ['control [[], 10, 2]', [[], 10, 2], []], ['control [[[39, 3, "ce"], [20, 10, "ce"], [16, 1, "ce"], [20, 10, "ce"], [22...', [[[39, 3, 'ce'], [20, 10, 'ce'], [16, 1, 'ce'], [20, 10, 'ce'], [22, 3, 'ue'], [40, 10, 'ce'], [15, 10, 'ce']], 5, 3], [[3, 22]]]], [['regression [[[32, 2, "ue"], [1, 1, "ue"], [28, 9, "ce"], [20, 10, "ce"], [13, ...', [[[32, 2, 'ue'], [1, 1, 'ue'], [28, 9, 'ce'], [20, 10, 'ce'], [13, 10, 'ce'], [28, 10, 'ce'], [6, 9, 'ce'], [36, 3, 'ce']], 15, 2], [[1, 1], [2, 32], [10, 20]]], ['regression [[[3, 1, "ce"], [1, 1, "ce"], [2, 4, "ce"]], 5, 2]', [[[3, 1, 'ce'], [1, 1, 'ce'], [2, 4, 'ce']], 5, 2], [[1, 3]]], ['control [[[39, 3, "ce"], [20, 10, "ce"], [16, 1, "ce"], [20, 10, "ce"], [22...', [[[39, 3, 'ce'], [20, 10, 'ce'], [16, 1, 'ce'], [20, 10, 'ce'], [22, 3, 'ue'], [40, 10, 'ce'], [15, 10, 'ce']], 5, 3], [[3, 22]]], ['control [[[14, 10, "ce"], [33, 1, "ce"], [3, 9, "ce"], [11, 1, "ce"], [9, 2...', [[[14, 10, 'ce'], [33, 1, 'ce'], [3, 9, 'ce'], [11, 1, 'ce'], [9, 2, 'ce'], [7, 9, 'ce'], [29, 3, 'ce'], [27, 1, 'ce']], 10, 3], []], ['control [[[2, 10, "ue"], [35, 1, "ce"], [6, 3, "ue"], [37, 10, "ce"]], 10, 3]', [[[2, 10, 'ue'], [35, 1, 'ce'], [6, 3, 'ue'], [37, 10, 'ce']], 10, 3], [[3, 6], [10, 2]]], ['control [[[5, 9, "ce"], [3, 10, "ue"], [23, 1, "ce"]], 15, 3]', [[[5, 9, 'ce'], [3, 10, 'ue'], [23, 1, 'ce']], 15, 3], [[10, 3]]], ['control [[[38, 2, "ce"], [26, 10, "ce"], [5, 1, "ce"], [32, 2, "ce"]], 5, 3]', [[[38, 2, 'ce'], [26, 10, 'ce'], [5, 1, 'ce'], [32, 2, 'ce']], 5, 3], []], ['control [[[1, 10, "ce"], [11, 10, "ue"], [29, 10, "ce"]], 5, 2]', [[[1, 10, 'ce'], [11, 10, 'ue'], [29, 10, 'ce']], 5, 2], [[10, 11]]]]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
regression [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...[[1, 0], [2, 27]][[2, 0]]Failed
regression [[[35, 3, "ce"], [11, 1, "ce"], [8, 1, "ce"], [28, 3, "ce"], [3, 10...[[3, 23], [10, 3]][[3, 35], [10, 3]]Failed
control [[[39, 3, "ce"], [20, 10, "ce"], [16, 1, "ce"], [20, 10, "ce"], [22...[[3, 22]][[3, 22]]Passed
control [[[14, 10, "ce"], [33, 1, "ce"], [3, 9, "ce"], [11, 1, "ce"], [9, 2...[][]Passed
control [[[2, 10, "ue"], [35, 1, "ce"], [6, 3, "ue"], [37, 10, "ce"]], 10, 3][[3, 6], [10, 2]][[3, 6], [10, 2]]Passed
control [[[5, 9, "ce"], [3, 10, "ue"], [23, 1, "ce"]], 15, 3][[10, 3]][[10, 3]]Passed
control [[[38, 2, "ce"], [26, 10, "ce"], [5, 1, "ce"], [32, 2, "ce"]], 5, 3][][]Passed
control [[[1, 10, "ce"], [11, 10, "ue"], [29, 10, "ce"]], 5, 2][[10, 11]][[10, 11]]Passed

SHA-256 / 631a9b4d73c5b6ec85089a7fd5093c7244344f02ed22eecd495225fd33da45d2

3 / The verified repair

Exit 0
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(events, window, threshold):
    retired = {}
    hist = {}
    for t, row, kind in sorted(events, key=lambda e: e[0]):
        if row in retired:
            continue
        if kind == 'ue':
            retired[row] = t
            continue
        h = [x for x in hist.get(row, []) if x > t - window] + [t]
        hist[row] = h
        if len(h) >= threshold:
            retired[row] = t
    return sorted([rw, tm] for rw, tm in retired.items())
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[['regression [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...', [[[21, 9, 'ce'], [27, 2, 'ue'], [0, 2, 'ue'], [21, 1, 'ce'], [9, 10, 'ce'], [0, 1, 'ce']], 5, 2], [[2, 0]]], ['regression [[[35, 3, "ce"], [11, 1, "ce"], [8, 1, "ce"], [28, 3, "ce"], [3, 10...', [[[35, 3, 'ce'], [11, 1, 'ce'], [8, 1, 'ce'], [28, 3, 'ce'], [3, 10, 'ue'], [23, 3, 'ce'], [13, 10, 'ce']], 15, 3], [[3, 35], [10, 3]]], ['control [[[39, 3, "ce"], [20, 10, "ce"], [16, 1, "ce"], [20, 10, "ce"], [22...', [[[39, 3, 'ce'], [20, 10, 'ce'], [16, 1, 'ce'], [20, 10, 'ce'], [22, 3, 'ue'], [40, 10, 'ce'], [15, 10, 'ce']], 5, 3], [[3, 22]]], ['control [[[14, 10, "ce"], [33, 1, "ce"], [3, 9, "ce"], [11, 1, "ce"], [9, 2...', [[[14, 10, 'ce'], [33, 1, 'ce'], [3, 9, 'ce'], [11, 1, 'ce'], [9, 2, 'ce'], [7, 9, 'ce'], [29, 3, 'ce'], [27, 1, 'ce']], 10, 3], []], ['control [[[2, 10, "ue"], [35, 1, "ce"], [6, 3, "ue"], [37, 10, "ce"]], 10, 3]', [[[2, 10, 'ue'], [35, 1, 'ce'], [6, 3, 'ue'], [37, 10, 'ce']], 10, 3], [[3, 6], [10, 2]]], ['control [[[5, 9, "ce"], [3, 10, "ue"], [23, 1, "ce"]], 15, 3]', [[[5, 9, 'ce'], [3, 10, 'ue'], [23, 1, 'ce']], 15, 3], [[10, 3]]], ['control [[[38, 2, "ce"], [26, 10, "ce"], [5, 1, "ce"], [32, 2, "ce"]], 5, 3]', [[[38, 2, 'ce'], [26, 10, 'ce'], [5, 1, 'ce'], [32, 2, 'ce']], 5, 3], []], ['control [[[1, 10, "ce"], [11, 10, "ue"], [29, 10, "ce"]], 5, 2]', [[[1, 10, 'ce'], [11, 10, 'ue'], [29, 10, 'ce']], 5, 2], [[10, 11]]]], [['regression [[[27, 3, "ce"], [2, 1, "ce"], [8, 1, "ce"], [28, 10, "ce"], [25, 1...', [[[27, 3, 'ce'], [2, 1, 'ce'], [8, 1, 'ce'], [28, 10, 'ce'], [25, 1, 'ce'], [19, 10, 'ce'], [8, 2, 'ue'], [7, 10, 'ce']], 5, 2], [[2, 8]]], ['regression [[[20, 1, "ce"], [37, 1, "ue"], [38, 10, "ce"], [24, 1, "ce"], [28,...', [[[20, 1, 'ce'], [37, 1, 'ue'], [38, 10, 'ce'], [24, 1, 'ce'], [28, 10, 'ce'], [7, 9, 'ce'], [7, 2, 'ce'], [22, 9, 'ce'], [31, 10, 'ce']], 10, 3], [[1, 37]]], ['control [[[1, 10, "ce"], [11, 10, "ue"], [29, 10, "ce"]], 5, 2]', [[[1, 10, 'ce'], [11, 10, 'ue'], [29, 10, 'ce']], 5, 2], [[10, 11]]], ['control [[[3, 3, "ue"], [24, 2, "ce"], [16, 1, "ue"], [12, 3, "ce"], [18, 3...', [[[3, 3, 'ue'], [24, 2, 'ce'], [16, 1, 'ue'], [12, 3, 'ce'], [18, 3, 'ce'], [3, 1, 'ce'], [2, 3, 'ce']], 5, 3], [[1, 16], [3, 3]]], ['control [[[32, 2, "ce"], [16, 10, "ue"], [26, 1, "ue"], [3, 10, "ce"], [1, ...', [[[32, 2, 'ce'], [16, 10, 'ue'], [26, 1, 'ue'], [3, 10, 'ce'], [1, 2, 'ce']], 10, 3], [[1, 26], [10, 16]]], ['control [[[15, 3, "ce"], [35, 9, "ce"], [0, 3, "ue"]], 5, 2]', [[[15, 3, 'ce'], [35, 9, 'ce'], [0, 3, 'ue']], 5, 2], [[3, 0]]], ['control [[[3, 2, "ue"], [11, 1, "ce"], [29, 1, "ce"], [28, 3, "ce"]], 5, 2]', [[[3, 2, 'ue'], [11, 1, 'ce'], [29, 1, 'ce'], [28, 3, 'ce']], 5, 2], [[2, 3]]], ['control [[[2, 3, "ce"], [35, 9, "ce"], [27, 2, "ce"], [40, 10, "ce"], [38, ...', [[[2, 3, 'ce'], [35, 9, 'ce'], [27, 2, 'ce'], [40, 10, 'ce'], [38, 3, 'ce'], [23, 3, 'ue'], [7, 10, 'ce'], [29, 9, 'ue']], 15, 3], [[3, 23], [9, 29]]]], [['regression [[[3, 1, "ce"], [1, 1, "ce"], [2, 4, "ce"]], 5, 2]', [[[3, 1, 'ce'], [1, 1, 'ce'], [2, 4, 'ce']], 5, 2], [[1, 3]]], ['regression [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...', [[[21, 9, 'ce'], [27, 2, 'ue'], [0, 2, 'ue'], [21, 1, 'ce'], [9, 10, 'ce'], [0, 1, 'ce']], 5, 2], [[2, 0]]], ['control [[[2, 3, "ce"], [35, 9, "ce"], [27, 2, "ce"], [40, 10, "ce"], [38, ...', [[[2, 3, 'ce'], [35, 9, 'ce'], [27, 2, 'ce'], [40, 10, 'ce'], [38, 3, 'ce'], [23, 3, 'ue'], [7, 10, 'ce'], [29, 9, 'ue']], 15, 3], [[3, 23], [9, 29]]], ['control [[[12, 3, "ce"], [37, 2, "ce"], [33, 1, "ce"], [12, 10, "ce"], [4, ...', [[[12, 3, 'ce'], [37, 2, 'ce'], [33, 1, 'ce'], [12, 10, 'ce'], [4, 3, 'ce'], [7, 10, 'ce'], [34, 1, 'ue'], [1, 1, 'ce'], [11, 3, 'ce']], 5, 3], [[1, 34]]], ['control [[[26, 9, "ce"], [40, 3, "ce"], [26, 10, "ce"], [40, 9, "ce"], [23,...', [[[26, 9, 'ce'], [40, 3, 'ce'], [26, 10, 'ce'], [40, 9, 'ce'], [23, 10, 'ue'], [9, 10, 'ce'], [21, 9, 'ce'], [0, 3, 'ce'], [29, 9, 'ue'], [8, 1, 'ce']], 5, 3], [[9, 29], [10, 23]]], ['control [[[39, 10, "ce"], [4, 3, "ce"], [20, 2, "ce"], [33, 1, "ce"]], 15, 3]', [[[39, 10, 'ce'], [4, 3, 'ce'], [20, 2, 'ce'], [33, 1, 'ce']], 15, 3], []], ['control [[[0, 1, "ce"], [10, 1, "ce"], [20, 1, "ce"]], 10, 2]', [[[0, 1, 'ce'], [10, 1, 'ce'], [20, 1, 'ce']], 10, 2], []], ['control [[[0, 1, "ce"], [9, 1, "ce"]], 10, 2]', [[[0, 1, 'ce'], [9, 1, 'ce']], 10, 2], [[1, 9]]]], [['regression [[[30, 1, "ce"], [30, 1, "ce"], [0, 3, "ce"], [16, 1, "ue"], [21, 3...', [[[30, 1, 'ce'], [30, 1, 'ce'], [0, 3, 'ce'], [16, 1, 'ue'], [21, 3, 'ue'], [33, 3, 'ce'], [30, 3, 'ue'], [29, 9, 'ue'], [1, 9, 'ce']], 15, 2], [[1, 16], [3, 21], [9, 29]]], ['regression [[[27, 3, "ce"], [2, 1, "ce"], [8, 1, "ce"], [28, 10, "ce"], [25, 1...', [[[27, 3, 'ce'], [2, 1, 'ce'], [8, 1, 'ce'], [28, 10, 'ce'], [25, 1, 'ce'], [19, 10, 'ce'], [8, 2, 'ue'], [7, 10, 'ce']], 5, 2], [[2, 8]]], ['control [[[0, 1, "ce"], [9, 1, "ce"]], 10, 2]', [[[0, 1, 'ce'], [9, 1, 'ce']], 10, 2], [[1, 9]]], ['control [[[5, 2, "ue"], [6, 2, "ce"], [7, 2, "ue"]], 10, 3]', [[[5, 2, 'ue'], [6, 2, 'ce'], [7, 2, 'ue']], 10, 3], [[2, 5]]], ['control [[[0, 1, "ce"], [1, 2, "ce"], [2, 3, "ce"]], 10, 2]', [[[0, 1, 'ce'], [1, 2, 'ce'], [2, 3, 'ce']], 10, 2], []], ['control [[[0, 8, "ce"], [1, 9, "ce"]], 10, 2]', [[[0, 8, 'ce'], [1, 9, 'ce']], 10, 2], []], ['control [[], 10, 2]', [[], 10, 2], []], ['control [[[39, 3, "ce"], [20, 10, "ce"], [16, 1, "ce"], [20, 10, "ce"], [22...', [[[39, 3, 'ce'], [20, 10, 'ce'], [16, 1, 'ce'], [20, 10, 'ce'], [22, 3, 'ue'], [40, 10, 'ce'], [15, 10, 'ce']], 5, 3], [[3, 22]]]], [['regression [[[32, 2, "ue"], [1, 1, "ue"], [28, 9, "ce"], [20, 10, "ce"], [13, ...', [[[32, 2, 'ue'], [1, 1, 'ue'], [28, 9, 'ce'], [20, 10, 'ce'], [13, 10, 'ce'], [28, 10, 'ce'], [6, 9, 'ce'], [36, 3, 'ce']], 15, 2], [[1, 1], [2, 32], [10, 20]]], ['regression [[[3, 1, "ce"], [1, 1, "ce"], [2, 4, "ce"]], 5, 2]', [[[3, 1, 'ce'], [1, 1, 'ce'], [2, 4, 'ce']], 5, 2], [[1, 3]]], ['control [[[39, 3, "ce"], [20, 10, "ce"], [16, 1, "ce"], [20, 10, "ce"], [22...', [[[39, 3, 'ce'], [20, 10, 'ce'], [16, 1, 'ce'], [20, 10, 'ce'], [22, 3, 'ue'], [40, 10, 'ce'], [15, 10, 'ce']], 5, 3], [[3, 22]]], ['control [[[14, 10, "ce"], [33, 1, "ce"], [3, 9, "ce"], [11, 1, "ce"], [9, 2...', [[[14, 10, 'ce'], [33, 1, 'ce'], [3, 9, 'ce'], [11, 1, 'ce'], [9, 2, 'ce'], [7, 9, 'ce'], [29, 3, 'ce'], [27, 1, 'ce']], 10, 3], []], ['control [[[2, 10, "ue"], [35, 1, "ce"], [6, 3, "ue"], [37, 10, "ce"]], 10, 3]', [[[2, 10, 'ue'], [35, 1, 'ce'], [6, 3, 'ue'], [37, 10, 'ce']], 10, 3], [[3, 6], [10, 2]]], ['control [[[5, 9, "ce"], [3, 10, "ue"], [23, 1, "ce"]], 15, 3]', [[[5, 9, 'ce'], [3, 10, 'ue'], [23, 1, 'ce']], 15, 3], [[10, 3]]], ['control [[[38, 2, "ce"], [26, 10, "ce"], [5, 1, "ce"], [32, 2, "ce"]], 5, 3]', [[[38, 2, 'ce'], [26, 10, 'ce'], [5, 1, 'ce'], [32, 2, 'ce']], 5, 3], []], ['control [[[1, 10, "ce"], [11, 10, "ue"], [29, 10, "ce"]], 5, 2]', [[[1, 10, 'ce'], [11, 10, 'ue'], [29, 10, 'ce']], 5, 2], [[10, 11]]]]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
regression [[[21, 9, "ce"], [27, 2, "ue"], [0, 2, "ue"], [21, 1, "ce"], [9, 10...[[2, 0]][[2, 0]]Passed
regression [[[35, 3, "ce"], [11, 1, "ce"], [8, 1, "ce"], [28, 3, "ce"], [3, 10...[[3, 35], [10, 3]][[3, 35], [10, 3]]Passed
control [[[39, 3, "ce"], [20, 10, "ce"], [16, 1, "ce"], [20, 10, "ce"], [22...[[3, 22]][[3, 22]]Passed
control [[[14, 10, "ce"], [33, 1, "ce"], [3, 9, "ce"], [11, 1, "ce"], [9, 2...[][]Passed
control [[[2, 10, "ue"], [35, 1, "ce"], [6, 3, "ue"], [37, 10, "ce"]], 10, 3][[3, 6], [10, 2]][[3, 6], [10, 2]]Passed
control [[[5, 9, "ce"], [3, 10, "ue"], [23, 1, "ce"]], 15, 3][[10, 3]][[10, 3]]Passed
control [[[38, 2, "ce"], [26, 10, "ce"], [5, 1, "ce"], [32, 2, "ce"]], 5, 3][][]Passed
control [[[1, 10, "ce"], [11, 10, "ue"], [29, 10, "ce"]], 5, 2][[10, 11]][[10, 11]]Passed

SHA-256 / 907f5103e384e91aa1df6bad99769d2f0a872b4bbe55e18e999889203538e2dd

Verification & scope

A deterministic, bounded teaching model of the named code under the stated contract; not a production codec. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:36.468183+00:00.

Case digest / fc95ed247852f3319226bd6c74dcf6f6c50e9f20f1788d55b12b9459df2455ba