FA-74201 / Feature flag rollout bucketing / Open access
Guarded progressive ramp: Healthy intervals resume a rolled-back ramp · case 01
After an automatic rollback the ramp quietly climbs back up without human review.
ROOT CAUSE
Nothing short-circuits later checks once the state is rolled_back.
VERIFIED REPAIR
Keep the ramp at 0 for every check after a rollback.
Unsuccessful approach: Resetting to the first stage on the next check still resumes the ramp automatically.
Case contract
stages is an increasing list of percentages; the ramp starts at stages[0]. Each check [error_rate, sample] is processed in order: after a rollback everything stays at 0; a sample below 100 holds the current stage; an error rate above 0.02 rolls back to 0 permanently; otherwise advance one stage (staying at the last). State is complete at the last stage, rolled_back after a rollback, else ramping. Return [final percent, state, percent after each check].
Why this case matters
Automated ramps with guardrails must neither overreact to noise nor resume after a rollback.
1 / The failure
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(stages, checks):
idx = 0
history = []
state = 'ramping' if len(stages) > 1 else 'complete'
for rate, sample in checks:
if sample < 100:
history.append(stages[idx])
continue
if rate > 0.02:
state = 'rolled_back'
history.append(0)
continue
idx = min(idx + 1, len(stages) - 1)
if idx == len(stages) - 1:
state = 'complete'
history.append(stages[idx])
final = 0 if state == 'rolled_back' else stages[idx]
return [final, state, history]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('low-sample breach only holds', [[1, 5, 25, 100], [[0.5, 20], [0, 500]]], [5, 'ramping', [1, 5]]),
('rollback is terminal',
[[1, 5, 25, 100], [[0.05, 500], [0, 500], [0, 500]]],
[0, 'rolled_back', [0, 0, 0]]),
('error rate exactly at threshold advances', [[10, 50, 100], [[0.02, 500]]], [50, 'ramping', [50]]),
('error rate just above threshold rolls back', [[10, 50, 100], [[0.024, 500]]], [0, 'rolled_back', [0]]),
('history records the stage after advancing',
[[1, 5, 25, 100], [[0, 500], [0, 500]]],
[25, 'ramping', [5, 25]]),
('guardrail sample 1', [[1, 5, 25, 100], [[0.01, 50], [0.02, 50], [0.024, 50]]], [1, 'ramping', [1, 1, 1]]),
('guardrail sample 2',
[[1, 5, 25, 100], [[0.024, 100], [0.01, 99], [0.05, 500]]],
[0, 'rolled_back', [0, 0, 0]]),
('guardrail sample 3', [[1, 5, 25, 100], [[0.05, 99], [0.05, 500]]], [0, 'rolled_back', [1, 0]])],
[('rollback is terminal',
[[1, 5, 25, 100], [[0.05, 500], [0, 500], [0, 500]]],
[0, 'rolled_back', [0, 0, 0]]),
('error rate exactly at threshold advances', [[10, 50, 100], [[0.02, 500]]], [50, 'ramping', [50]]),
('error rate just above threshold rolls back', [[10, 50, 100], [[0.024, 500]]], [0, 'rolled_back', [0]]),
('history records the stage after advancing',
[[1, 5, 25, 100], [[0, 500], [0, 500]]],
[25, 'ramping', [5, 25]]),
('final percent after rollback is zero', [[5, 100], [[0, 500], [0.1, 500]]], [0, 'rolled_back', [100, 0]]),
('guardrail sample 6',
[[100], [[0.024, 99], [0.01, 99], [0.02, 500], [0.024, 500], [0.01, 50], [0.024, 100]]],
[0, 'rolled_back', [100, 100, 100, 0, 0, 0]]),
('guardrail sample 7',
[[10, 50, 100], [[0, 50], [0, 50], [0.05, 99], [0.02, 50]]],
[10, 'ramping', [10, 10, 10, 10]]),
('guardrail sample 19',
[[5, 100], [[0, 100], [0.05, 50], [0.024, 500], [0.01, 100], [0.024, 50]]],
[0, 'rolled_back', [100, 100, 0, 0, 0]])],
[('error rate exactly at threshold advances', [[10, 50, 100], [[0.02, 500]]], [50, 'ramping', [50]]),
('error rate just above threshold rolls back', [[10, 50, 100], [[0.024, 500]]], [0, 'rolled_back', [0]]),
('history records the stage after advancing',
[[1, 5, 25, 100], [[0, 500], [0, 500]]],
[25, 'ramping', [5, 25]]),
('final percent after rollback is zero', [[5, 100], [[0, 500], [0.1, 500]]], [0, 'rolled_back', [100, 0]]),
('single stage is complete from the start', [[100], [[0, 50]]], [100, 'complete', [100]]),
('guardrail sample 11',
[[100], [[0.05, 100], [0.024, 50], [0.02, 50], [0.02, 99], [0.05, 50], [0.05, 50]]],
[0, 'rolled_back', [0, 0, 0, 0, 0, 0]]),
('guardrail sample 12', [[10, 50, 100], [[0.02, 100]]], [50, 'ramping', [50]]),
('guardrail sample 39',
[[10, 50, 100], [[0, 500], [0.024, 500], [0.02, 99], [0.024, 99]]],
[0, 'rolled_back', [50, 0, 0, 0]])],
[('error rate just above threshold rolls back', [[10, 50, 100], [[0.024, 500]]], [0, 'rolled_back', [0]]),
('history records the stage after advancing',
[[1, 5, 25, 100], [[0, 500], [0, 500]]],
[25, 'ramping', [5, 25]]),
('final percent after rollback is zero', [[5, 100], [[0, 500], [0.1, 500]]], [0, 'rolled_back', [100, 0]]),
('single stage is complete from the start', [[100], [[0, 50]]], [100, 'complete', [100]]),
('rollback after completion',
[[10, 50, 100], [[0, 500], [0, 500], [0.03, 500]]],
[0, 'rolled_back', [50, 100, 0]]),
('guardrail sample 16',
[[5, 100], [[0.01, 50], [0.02, 50], [0.024, 100], [0.02, 50], [0.024, 500]]],
[0, 'rolled_back', [5, 5, 0, 0, 0]]),
('guardrail sample 17', [[5, 100], [[0.05, 500], [0.01, 99]]], [0, 'rolled_back', [0, 0]]),
('guardrail sample 57',
[[1, 5, 25, 100], [[0.024, 500], [0.024, 99], [0.02, 99]]],
[0, 'rolled_back', [0, 0, 0]])],
[('low-sample breach only holds', [[1, 5, 25, 100], [[0.5, 20], [0, 500]]], [5, 'ramping', [1, 5]]),
('history records the stage after advancing',
[[1, 5, 25, 100], [[0, 500], [0, 500]]],
[25, 'ramping', [5, 25]]),
('final percent after rollback is zero', [[5, 100], [[0, 500], [0.1, 500]]], [0, 'rolled_back', [100, 0]]),
('single stage is complete from the start', [[100], [[0, 50]]], [100, 'complete', [100]]),
('rollback after completion',
[[10, 50, 100], [[0, 500], [0, 500], [0.03, 500]]],
[0, 'rolled_back', [50, 100, 0]]),
('guardrail sample 17', [[5, 100], [[0.05, 500], [0.01, 99]]], [0, 'rolled_back', [0, 0]]),
('guardrail sample 21',
[[5, 100], [[0.024, 99], [0, 100], [0.02, 99], [0.01, 50], [0.02, 50], [0.02, 100]]],
[100, 'complete', [5, 100, 100, 100, 100, 100]]),
('guardrail sample 22',
[[100], [[0.01, 100], [0.05, 99], [0.01, 100], [0.05, 100], [0, 500], [0.01, 50]]],
[0, 'rolled_back', [100, 100, 100, 0, 0, 0]])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| low-sample breach only holds | [5, 'ramping', [1, 5]] | [5, 'ramping', [1, 5]] | Passed |
| rollback is terminal | [0, 'rolled_back', [0, 5, 25]] | [0, 'rolled_back', [0, 0, 0]] | Failed |
| error rate exactly at threshold advances | [50, 'ramping', [50]] | [50, 'ramping', [50]] | Passed |
| error rate just above threshold rolls back | [0, 'rolled_back', [0]] | [0, 'rolled_back', [0]] | Passed |
| history records the stage after advancing | [25, 'ramping', [5, 25]] | [25, 'ramping', [5, 25]] | Passed |
| guardrail sample 1 | [1, 'ramping', [1, 1, 1]] | [1, 'ramping', [1, 1, 1]] | Passed |
| guardrail sample 2 | [0, 'rolled_back', [0, 1, 0]] | [0, 'rolled_back', [0, 0, 0]] | Failed |
| guardrail sample 3 | [0, 'rolled_back', [1, 0]] | [0, 'rolled_back', [1, 0]] | Passed |
SHA-256 / 18ff97c3404f4778b12e887982b1985017760ec4e90b29902f3b8d8015d5b7c5
2 / The unsuccessful fix
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(stages, checks):
idx = 0
history = []
state = 'ramping' if len(stages) > 1 else 'complete'
for rate, sample in checks:
if state == 'rolled_back':
state = 'ramping'
idx = 0
if sample < 100:
history.append(stages[idx])
continue
if rate > 0.02:
state = 'rolled_back'
history.append(0)
continue
idx = min(idx + 1, len(stages) - 1)
if idx == len(stages) - 1:
state = 'complete'
history.append(stages[idx])
final = 0 if state == 'rolled_back' else stages[idx]
return [final, state, history]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('low-sample breach only holds', [[1, 5, 25, 100], [[0.5, 20], [0, 500]]], [5, 'ramping', [1, 5]]),
('rollback is terminal',
[[1, 5, 25, 100], [[0.05, 500], [0, 500], [0, 500]]],
[0, 'rolled_back', [0, 0, 0]]),
('error rate exactly at threshold advances', [[10, 50, 100], [[0.02, 500]]], [50, 'ramping', [50]]),
('error rate just above threshold rolls back', [[10, 50, 100], [[0.024, 500]]], [0, 'rolled_back', [0]]),
('history records the stage after advancing',
[[1, 5, 25, 100], [[0, 500], [0, 500]]],
[25, 'ramping', [5, 25]]),
('guardrail sample 1', [[1, 5, 25, 100], [[0.01, 50], [0.02, 50], [0.024, 50]]], [1, 'ramping', [1, 1, 1]]),
('guardrail sample 2',
[[1, 5, 25, 100], [[0.024, 100], [0.01, 99], [0.05, 500]]],
[0, 'rolled_back', [0, 0, 0]]),
('guardrail sample 3', [[1, 5, 25, 100], [[0.05, 99], [0.05, 500]]], [0, 'rolled_back', [1, 0]])],
[('rollback is terminal',
[[1, 5, 25, 100], [[0.05, 500], [0, 500], [0, 500]]],
[0, 'rolled_back', [0, 0, 0]]),
('error rate exactly at threshold advances', [[10, 50, 100], [[0.02, 500]]], [50, 'ramping', [50]]),
('error rate just above threshold rolls back', [[10, 50, 100], [[0.024, 500]]], [0, 'rolled_back', [0]]),
('history records the stage after advancing',
[[1, 5, 25, 100], [[0, 500], [0, 500]]],
[25, 'ramping', [5, 25]]),
('final percent after rollback is zero', [[5, 100], [[0, 500], [0.1, 500]]], [0, 'rolled_back', [100, 0]]),
('guardrail sample 6',
[[100], [[0.024, 99], [0.01, 99], [0.02, 500], [0.024, 500], [0.01, 50], [0.024, 100]]],
[0, 'rolled_back', [100, 100, 100, 0, 0, 0]]),
('guardrail sample 7',
[[10, 50, 100], [[0, 50], [0, 50], [0.05, 99], [0.02, 50]]],
[10, 'ramping', [10, 10, 10, 10]]),
('guardrail sample 19',
[[5, 100], [[0, 100], [0.05, 50], [0.024, 500], [0.01, 100], [0.024, 50]]],
[0, 'rolled_back', [100, 100, 0, 0, 0]])],
[('error rate exactly at threshold advances', [[10, 50, 100], [[0.02, 500]]], [50, 'ramping', [50]]),
('error rate just above threshold rolls back', [[10, 50, 100], [[0.024, 500]]], [0, 'rolled_back', [0]]),
('history records the stage after advancing',
[[1, 5, 25, 100], [[0, 500], [0, 500]]],
[25, 'ramping', [5, 25]]),
('final percent after rollback is zero', [[5, 100], [[0, 500], [0.1, 500]]], [0, 'rolled_back', [100, 0]]),
('single stage is complete from the start', [[100], [[0, 50]]], [100, 'complete', [100]]),
('guardrail sample 11',
[[100], [[0.05, 100], [0.024, 50], [0.02, 50], [0.02, 99], [0.05, 50], [0.05, 50]]],
[0, 'rolled_back', [0, 0, 0, 0, 0, 0]]),
('guardrail sample 12', [[10, 50, 100], [[0.02, 100]]], [50, 'ramping', [50]]),
('guardrail sample 39',
[[10, 50, 100], [[0, 500], [0.024, 500], [0.02, 99], [0.024, 99]]],
[0, 'rolled_back', [50, 0, 0, 0]])],
[('error rate just above threshold rolls back', [[10, 50, 100], [[0.024, 500]]], [0, 'rolled_back', [0]]),
('history records the stage after advancing',
[[1, 5, 25, 100], [[0, 500], [0, 500]]],
[25, 'ramping', [5, 25]]),
('final percent after rollback is zero', [[5, 100], [[0, 500], [0.1, 500]]], [0, 'rolled_back', [100, 0]]),
('single stage is complete from the start', [[100], [[0, 50]]], [100, 'complete', [100]]),
('rollback after completion',
[[10, 50, 100], [[0, 500], [0, 500], [0.03, 500]]],
[0, 'rolled_back', [50, 100, 0]]),
('guardrail sample 16',
[[5, 100], [[0.01, 50], [0.02, 50], [0.024, 100], [0.02, 50], [0.024, 500]]],
[0, 'rolled_back', [5, 5, 0, 0, 0]]),
('guardrail sample 17', [[5, 100], [[0.05, 500], [0.01, 99]]], [0, 'rolled_back', [0, 0]]),
('guardrail sample 57',
[[1, 5, 25, 100], [[0.024, 500], [0.024, 99], [0.02, 99]]],
[0, 'rolled_back', [0, 0, 0]])],
[('low-sample breach only holds', [[1, 5, 25, 100], [[0.5, 20], [0, 500]]], [5, 'ramping', [1, 5]]),
('history records the stage after advancing',
[[1, 5, 25, 100], [[0, 500], [0, 500]]],
[25, 'ramping', [5, 25]]),
('final percent after rollback is zero', [[5, 100], [[0, 500], [0.1, 500]]], [0, 'rolled_back', [100, 0]]),
('single stage is complete from the start', [[100], [[0, 50]]], [100, 'complete', [100]]),
('rollback after completion',
[[10, 50, 100], [[0, 500], [0, 500], [0.03, 500]]],
[0, 'rolled_back', [50, 100, 0]]),
('guardrail sample 17', [[5, 100], [[0.05, 500], [0.01, 99]]], [0, 'rolled_back', [0, 0]]),
('guardrail sample 21',
[[5, 100], [[0.024, 99], [0, 100], [0.02, 99], [0.01, 50], [0.02, 50], [0.02, 100]]],
[100, 'complete', [5, 100, 100, 100, 100, 100]]),
('guardrail sample 22',
[[100], [[0.01, 100], [0.05, 99], [0.01, 100], [0.05, 100], [0, 500], [0.01, 50]]],
[0, 'rolled_back', [100, 100, 100, 0, 0, 0]])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| low-sample breach only holds | [5, 'ramping', [1, 5]] | [5, 'ramping', [1, 5]] | Passed |
| rollback is terminal | [25, 'ramping', [0, 5, 25]] | [0, 'rolled_back', [0, 0, 0]] | Failed |
| error rate exactly at threshold advances | [50, 'ramping', [50]] | [50, 'ramping', [50]] | Passed |
| error rate just above threshold rolls back | [0, 'rolled_back', [0]] | [0, 'rolled_back', [0]] | Passed |
| history records the stage after advancing | [25, 'ramping', [5, 25]] | [25, 'ramping', [5, 25]] | Passed |
| guardrail sample 1 | [1, 'ramping', [1, 1, 1]] | [1, 'ramping', [1, 1, 1]] | Passed |
| guardrail sample 2 | [0, 'rolled_back', [0, 1, 0]] | [0, 'rolled_back', [0, 0, 0]] | Failed |
| guardrail sample 3 | [0, 'rolled_back', [1, 0]] | [0, 'rolled_back', [1, 0]] | Passed |
SHA-256 / be649478cd5f98bf41ae013c2af1b93ed8050fdba66fb4cce8dc0a8051d3edcd
3 / The verified repair
Exit 0"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(stages, checks):
idx = 0
history = []
state = 'ramping' if len(stages) > 1 else 'complete'
for rate, sample in checks:
if state == 'rolled_back':
history.append(0)
continue
if sample < 100:
history.append(stages[idx])
continue
if rate > 0.02:
state = 'rolled_back'
history.append(0)
continue
idx = min(idx + 1, len(stages) - 1)
if idx == len(stages) - 1:
state = 'complete'
history.append(stages[idx])
final = 0 if state == 'rolled_back' else stages[idx]
return [final, state, history]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('low-sample breach only holds', [[1, 5, 25, 100], [[0.5, 20], [0, 500]]], [5, 'ramping', [1, 5]]),
('rollback is terminal',
[[1, 5, 25, 100], [[0.05, 500], [0, 500], [0, 500]]],
[0, 'rolled_back', [0, 0, 0]]),
('error rate exactly at threshold advances', [[10, 50, 100], [[0.02, 500]]], [50, 'ramping', [50]]),
('error rate just above threshold rolls back', [[10, 50, 100], [[0.024, 500]]], [0, 'rolled_back', [0]]),
('history records the stage after advancing',
[[1, 5, 25, 100], [[0, 500], [0, 500]]],
[25, 'ramping', [5, 25]]),
('guardrail sample 1', [[1, 5, 25, 100], [[0.01, 50], [0.02, 50], [0.024, 50]]], [1, 'ramping', [1, 1, 1]]),
('guardrail sample 2',
[[1, 5, 25, 100], [[0.024, 100], [0.01, 99], [0.05, 500]]],
[0, 'rolled_back', [0, 0, 0]]),
('guardrail sample 3', [[1, 5, 25, 100], [[0.05, 99], [0.05, 500]]], [0, 'rolled_back', [1, 0]])],
[('rollback is terminal',
[[1, 5, 25, 100], [[0.05, 500], [0, 500], [0, 500]]],
[0, 'rolled_back', [0, 0, 0]]),
('error rate exactly at threshold advances', [[10, 50, 100], [[0.02, 500]]], [50, 'ramping', [50]]),
('error rate just above threshold rolls back', [[10, 50, 100], [[0.024, 500]]], [0, 'rolled_back', [0]]),
('history records the stage after advancing',
[[1, 5, 25, 100], [[0, 500], [0, 500]]],
[25, 'ramping', [5, 25]]),
('final percent after rollback is zero', [[5, 100], [[0, 500], [0.1, 500]]], [0, 'rolled_back', [100, 0]]),
('guardrail sample 6',
[[100], [[0.024, 99], [0.01, 99], [0.02, 500], [0.024, 500], [0.01, 50], [0.024, 100]]],
[0, 'rolled_back', [100, 100, 100, 0, 0, 0]]),
('guardrail sample 7',
[[10, 50, 100], [[0, 50], [0, 50], [0.05, 99], [0.02, 50]]],
[10, 'ramping', [10, 10, 10, 10]]),
('guardrail sample 19',
[[5, 100], [[0, 100], [0.05, 50], [0.024, 500], [0.01, 100], [0.024, 50]]],
[0, 'rolled_back', [100, 100, 0, 0, 0]])],
[('error rate exactly at threshold advances', [[10, 50, 100], [[0.02, 500]]], [50, 'ramping', [50]]),
('error rate just above threshold rolls back', [[10, 50, 100], [[0.024, 500]]], [0, 'rolled_back', [0]]),
('history records the stage after advancing',
[[1, 5, 25, 100], [[0, 500], [0, 500]]],
[25, 'ramping', [5, 25]]),
('final percent after rollback is zero', [[5, 100], [[0, 500], [0.1, 500]]], [0, 'rolled_back', [100, 0]]),
('single stage is complete from the start', [[100], [[0, 50]]], [100, 'complete', [100]]),
('guardrail sample 11',
[[100], [[0.05, 100], [0.024, 50], [0.02, 50], [0.02, 99], [0.05, 50], [0.05, 50]]],
[0, 'rolled_back', [0, 0, 0, 0, 0, 0]]),
('guardrail sample 12', [[10, 50, 100], [[0.02, 100]]], [50, 'ramping', [50]]),
('guardrail sample 39',
[[10, 50, 100], [[0, 500], [0.024, 500], [0.02, 99], [0.024, 99]]],
[0, 'rolled_back', [50, 0, 0, 0]])],
[('error rate just above threshold rolls back', [[10, 50, 100], [[0.024, 500]]], [0, 'rolled_back', [0]]),
('history records the stage after advancing',
[[1, 5, 25, 100], [[0, 500], [0, 500]]],
[25, 'ramping', [5, 25]]),
('final percent after rollback is zero', [[5, 100], [[0, 500], [0.1, 500]]], [0, 'rolled_back', [100, 0]]),
('single stage is complete from the start', [[100], [[0, 50]]], [100, 'complete', [100]]),
('rollback after completion',
[[10, 50, 100], [[0, 500], [0, 500], [0.03, 500]]],
[0, 'rolled_back', [50, 100, 0]]),
('guardrail sample 16',
[[5, 100], [[0.01, 50], [0.02, 50], [0.024, 100], [0.02, 50], [0.024, 500]]],
[0, 'rolled_back', [5, 5, 0, 0, 0]]),
('guardrail sample 17', [[5, 100], [[0.05, 500], [0.01, 99]]], [0, 'rolled_back', [0, 0]]),
('guardrail sample 57',
[[1, 5, 25, 100], [[0.024, 500], [0.024, 99], [0.02, 99]]],
[0, 'rolled_back', [0, 0, 0]])],
[('low-sample breach only holds', [[1, 5, 25, 100], [[0.5, 20], [0, 500]]], [5, 'ramping', [1, 5]]),
('history records the stage after advancing',
[[1, 5, 25, 100], [[0, 500], [0, 500]]],
[25, 'ramping', [5, 25]]),
('final percent after rollback is zero', [[5, 100], [[0, 500], [0.1, 500]]], [0, 'rolled_back', [100, 0]]),
('single stage is complete from the start', [[100], [[0, 50]]], [100, 'complete', [100]]),
('rollback after completion',
[[10, 50, 100], [[0, 500], [0, 500], [0.03, 500]]],
[0, 'rolled_back', [50, 100, 0]]),
('guardrail sample 17', [[5, 100], [[0.05, 500], [0.01, 99]]], [0, 'rolled_back', [0, 0]]),
('guardrail sample 21',
[[5, 100], [[0.024, 99], [0, 100], [0.02, 99], [0.01, 50], [0.02, 50], [0.02, 100]]],
[100, 'complete', [5, 100, 100, 100, 100, 100]]),
('guardrail sample 22',
[[100], [[0.01, 100], [0.05, 99], [0.01, 100], [0.05, 100], [0, 500], [0.01, 50]]],
[0, 'rolled_back', [100, 100, 100, 0, 0, 0]])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| low-sample breach only holds | [5, 'ramping', [1, 5]] | [5, 'ramping', [1, 5]] | Passed |
| rollback is terminal | [0, 'rolled_back', [0, 0, 0]] | [0, 'rolled_back', [0, 0, 0]] | Passed |
| error rate exactly at threshold advances | [50, 'ramping', [50]] | [50, 'ramping', [50]] | Passed |
| error rate just above threshold rolls back | [0, 'rolled_back', [0]] | [0, 'rolled_back', [0]] | Passed |
| history records the stage after advancing | [25, 'ramping', [5, 25]] | [25, 'ramping', [5, 25]] | Passed |
| guardrail sample 1 | [1, 'ramping', [1, 1, 1]] | [1, 'ramping', [1, 1, 1]] | Passed |
| guardrail sample 2 | [0, 'rolled_back', [0, 0, 0]] | [0, 'rolled_back', [0, 0, 0]] | Passed |
| guardrail sample 3 | [0, 'rolled_back', [1, 0]] | [0, 'rolled_back', [1, 0]] | Passed |
SHA-256 / 2108669da4fec1b7f8e8147367dc36657fd50e63adcce0df997c825b35d348bc
Verification & scope
A deterministic toy flag-evaluation model with a stipulated contract; it does not reproduce any vendor SDK byte for byte. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.
Observations recorded using Python 3.12.14 at 2026-09-29T14:48:54.518653+00:00.
Case digest / a498f0cff60f114fe871fdba3a12aa7a19b1174e583856b906c4abf8ede7b205