FA-10311 / Observability / Open access

Alert evaluation state transition: Evaluation errors are mistaken for healthy results · case 01

Evaluation errors are mistaken for healthy results.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The implementation substitutes if error: return ('ok', 0) for if error: return ('error', count), so evaluation errors are mistaken for healthy results.

VERIFIED REPAIR

Preserve evaluation error state and prior count.

Unsuccessful approach: The attempted repair substitutes if error: return ('no-data', count). Fixture 2 still yields ('no-data', 2) instead of ('error', 2).

Case contract

Disabled rules become disabled; absent observations become no-data; explicit evaluation error becomes error; values above threshold become firing after required consecutive count, otherwise pending; healthy observations reset count to zero.

Why this case matters

An offline model of alert evaluation state transition, suitable for testing build and release tooling without external services.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(enabled, value, error, threshold, count, required):
    if not enabled: return ('disabled', 0)
    if error: return ('ok', 0)
    if value is None: return ('no-data', count)
    if value <= threshold: return ('ok', 0)
    count += 1
    return ('firing' if count >= required else 'pending', count)
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('fixture 1', solve(False, 9, False, 5, 2, 3), ('disabled', 0))
check('fixture 2', solve(True, 9, True, 5, 2, 3), ('error', 2))
check('fixture 3', solve(True, None, False, 5, 2, 3), ('no-data', 2))
check('fixture 4', solve(True, 5, False, 5, 2, 3), ('ok', 0))
check('fixture 5', solve(True, 4, False, 5, 2, 3), ('ok', 0))
check('fixture 6', solve(True, 6, False, 5, 1, 3), ('pending', 2))
check('fixture 7', solve(True, 6, False, 5, 2, 3), ('firing', 3))
check('fixture 8', solve(True, 6, False, 5, 3, 3), ('firing', 4))
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
fixture 1	`['disabled', 0]`	`['disabled', 0]`	Passed
fixture 2	`['ok', 0]`	`['error', 2]`	Failed
fixture 3	`['no-data', 2]`	`['no-data', 2]`	Passed
fixture 4	`['ok', 0]`	`['ok', 0]`	Passed
fixture 5	`['ok', 0]`	`['ok', 0]`	Passed
fixture 6	`['pending', 2]`	`['pending', 2]`	Passed
fixture 7	`['firing', 3]`	`['firing', 3]`	Passed
fixture 8	`['firing', 4]`	`['firing', 4]`	Passed

SHA-256 / 52758850c83105a1df0c1a98cf7e7d277214b7727da1c139e6c58765d84c1f00

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(enabled, value, error, threshold, count, required):
    if not enabled: return ('disabled', 0)
    if error: return ('no-data', count)
    if value is None: return ('no-data', count)
    if value <= threshold: return ('ok', 0)
    count += 1
    return ('firing' if count >= required else 'pending', count)
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('fixture 1', solve(False, 9, False, 5, 2, 3), ('disabled', 0))
check('fixture 2', solve(True, 9, True, 5, 2, 3), ('error', 2))
check('fixture 3', solve(True, None, False, 5, 2, 3), ('no-data', 2))
check('fixture 4', solve(True, 5, False, 5, 2, 3), ('ok', 0))
check('fixture 5', solve(True, 4, False, 5, 2, 3), ('ok', 0))
check('fixture 6', solve(True, 6, False, 5, 1, 3), ('pending', 2))
check('fixture 7', solve(True, 6, False, 5, 2, 3), ('firing', 3))
check('fixture 8', solve(True, 6, False, 5, 3, 3), ('firing', 4))
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
fixture 1	`['disabled', 0]`	`['disabled', 0]`	Passed
fixture 2	`['no-data', 2]`	`['error', 2]`	Failed
fixture 3	`['no-data', 2]`	`['no-data', 2]`	Passed
fixture 4	`['ok', 0]`	`['ok', 0]`	Passed
fixture 5	`['ok', 0]`	`['ok', 0]`	Passed
fixture 6	`['pending', 2]`	`['pending', 2]`	Passed
fixture 7	`['firing', 3]`	`['firing', 3]`	Passed
fixture 8	`['firing', 4]`	`['firing', 4]`	Passed

SHA-256 / a8fe982e39eab087dc8cd207c2b0acaba1fd184234c5aa96096273e3a4a183dd

3 / The verified repair

Exit 0

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(enabled, value, error, threshold, count, required):
    if not enabled: return ('disabled', 0)
    if error: return ('error', count)
    if value is None: return ('no-data', count)
    if value <= threshold: return ('ok', 0)
    count += 1
    return ('firing' if count >= required else 'pending', count)
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('fixture 1', solve(False, 9, False, 5, 2, 3), ('disabled', 0))
check('fixture 2', solve(True, 9, True, 5, 2, 3), ('error', 2))
check('fixture 3', solve(True, None, False, 5, 2, 3), ('no-data', 2))
check('fixture 4', solve(True, 5, False, 5, 2, 3), ('ok', 0))
check('fixture 5', solve(True, 4, False, 5, 2, 3), ('ok', 0))
check('fixture 6', solve(True, 6, False, 5, 1, 3), ('pending', 2))
check('fixture 7', solve(True, 6, False, 5, 2, 3), ('firing', 3))
check('fixture 8', solve(True, 6, False, 5, 3, 3), ('firing', 4))
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
fixture 1	`['disabled', 0]`	`['disabled', 0]`	Passed
fixture 2	`['error', 2]`	`['error', 2]`	Passed
fixture 3	`['no-data', 2]`	`['no-data', 2]`	Passed
fixture 4	`['ok', 0]`	`['ok', 0]`	Passed
fixture 5	`['ok', 0]`	`['ok', 0]`	Passed
fixture 6	`['pending', 2]`	`['pending', 2]`	Passed
fixture 7	`['firing', 3]`	`['firing', 3]`	Passed
fixture 8	`['firing', 4]`	`['firing', 4]`	Passed

SHA-256 / 35c4b559190c1ca2e050e8d7eb5ff1ba4510b8a78cb28644c58185ff701d959c

Verification & scope

This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:38:37.877042+00:00.

Case digest / 4208421580b7537f4a2b26bd4f607de16892e565f29a946ca3b503c4c4983bd7