{"abstract":"A guardrail whose lower bound equals the tolerated loss is accepted.","category":"Experiment statistics","checks":8,"contract":"primary is [lift, ci_low, ci_high]. Each guardrail [name, ci_low, ci_high, margin, direction] passes non-inferiority when higher_is_better and ci_low > -margin, or lower_is_better and ci_high < margin. Decision: no_ship if primary ci_high < 0; else if primary ci_low > 0 then \"blocked:<first failing guardrail in input order>\" or ship; else inconclusive. Return [decision, failing names in input order].","contract_signature":"primary, guardrails","evaluation_group":"w2-experiment-statistics-ship-decision","failed_approach":"Rounding the bound to two decimals first hides losses just past the margin.","family":"w2-experiment-statistics-ship-decision-margin-strictness","id":"FA-74631","implementations":{"attempt":{"sha256":"da8e97de013261a38631ce28e2ae3fe0d6616c507e6d565c3b4cf525e144b7a2","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(primary, guardrails):\n    failing = []\n    for name, lo, hi, margin, direction in guardrails:\n        if direction == 'higher_is_better':\n            ok = round(lo, 2) > -margin\n        else:\n            ok = hi < margin\n        if not ok:\n            failing.append(name)\n    if primary[2] < 0:\n        return ['no_ship', failing]\n    if primary[1] > 0:\n        if failing:\n            return ['blocked:' + failing[0], failing]\n        return ['ship', failing]\n    return ['inconclusive', failing]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('latency guardrail uses the upper bound',\n   [[0.02, 0.01, 0.03], [['latency', -0.01, 0.025, 0.02, 'lower_is_better']]],\n   ['blocked:latency', ['latency']]),\n  ('non-inferiority allows a small loss',\n   [[0.02, 0.01, 0.03], [['revenue', -0.015, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('readout sample 1',\n   [[0.02, 0.01, 0.04],\n    [['latency', -0.03, 0.0, 0.01, 'higher_is_better'],\n     ['crashes', -0.03, 0.020000000000000004, 0.01, 'lower_is_better']]],\n   ['blocked:latency', ['latency', 'crashes']]),\n  ('readout sample 2', [[-0.04, -0.05, -0.020000000000000004], []], ['no_ship', []]),\n  ('readout sample 3',\n   [[0.01, 0.0, 0.01],\n    [['revenue', 0.0, 0.03, 0.01, 'higher_is_better'],\n     ['latency', 0.004, 0.054000000000000006, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['latency']])],\n [('non-inferiority allows a small loss',\n   [[0.02, 0.01, 0.03], [['revenue', -0.015, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('readout sample 6',\n   [[0.01, 0.0, 0.01],\n    [['errors', -0.03, 0.0, 0.02, 'higher_is_better'],\n     ['crashes', 0.004, 0.014, 0.02, 'lower_is_better'],\n     ['revenue', -0.01, 0.019999999999999997, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['errors']]),\n  ('readout sample 7',\n   [[0.01, 0.0, 0.03],\n    [['latency', 0.004, 0.054000000000000006, 0.01, 'higher_is_better'],\n     ['errors', -0.03, 0.0, 0.02, 'higher_is_better'],\n     ['revenue', 0.004, 0.034, 0.02, 'higher_is_better']]],\n   ['inconclusive', ['errors']]),\n  ('readout sample 8',\n   [[-0.04, -0.05, -0.020000000000000004],\n    [['latency', -0.01, 0.019999999999999997, 0.01, 'lower_is_better'],\n     ['errors', 0.0, 0.03, 0.02, 'higher_is_better']]],\n   ['no_ship', ['latency']])],\n [('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('readout sample 11',\n   [[0.02, 0.01, 0.02], [['revenue', -0.01, 0.04, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('readout sample 12',\n   [[0.02, 0.01, 0.04], [['latency', -0.03, -0.019999999999999997, 0.01, 'lower_is_better']]],\n   ['ship', []]),\n  ('readout sample 13',\n   [[-0.04, -0.05, -0.04],\n    [['latency', -0.03, 0.0, 0.02, 'lower_is_better'],\n     ['revenue', 0.0, 0.03, 0.01, 'higher_is_better'],\n     ['crashes', -0.01, 0.0, 0.02, 'higher_is_better']]],\n   ['no_ship', []])],\n [('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('significant loss is no ship',\n   [[-0.02, -0.03, -0.01], [['latency', 0.0, 0.1, 0.02, 'lower_is_better']]],\n   ['no_ship', ['latency']]),\n  ('readout sample 16',\n   [[-0.01, -0.02, 0.009999999999999998],\n    [['latency', 0.004, 0.014, 0.02, 'higher_is_better'],\n     ['errors', -0.03, 0.020000000000000004, 0.01, 'lower_is_better'],\n     ['crashes', 0.004, 0.054000000000000006, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['errors', 'crashes']]),\n  ('readout sample 17',\n   [[-0.04, -0.05, -0.04], [['errors', 0.0, 0.05, 0.01, 'lower_is_better']]],\n   ['no_ship', ['errors']])],\n [('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('significant loss is no ship',\n   [[-0.02, -0.03, -0.01], [['latency', 0.0, 0.1, 0.02, 'lower_is_better']]],\n   ['no_ship', ['latency']]),\n  ('guardrail loss beyond margin',\n   [[0.02, 0.01, 0.03], [['revenue', -0.03, 0.0, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('readout sample 21',\n   [[0.01, 0.0, 0.03], [['latency', -0.02, 0.030000000000000002, 0.01, 'lower_is_better']]],\n   ['inconclusive', ['latency']])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"broken":{"sha256":"b4352f351e3af50008a9ab66c3bb418246d28951e570bae3464c5093b8eb7e26","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(primary, guardrails):\n    failing = []\n    for name, lo, hi, margin, direction in guardrails:\n        if direction == 'higher_is_better':\n            ok = lo >= -margin\n        else:\n            ok = hi < margin\n        if not ok:\n            failing.append(name)\n    if primary[2] < 0:\n        return ['no_ship', failing]\n    if primary[1] > 0:\n        if failing:\n            return ['blocked:' + failing[0], failing]\n        return ['ship', failing]\n    return ['inconclusive', failing]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('latency guardrail uses the upper bound',\n   [[0.02, 0.01, 0.03], [['latency', -0.01, 0.025, 0.02, 'lower_is_better']]],\n   ['blocked:latency', ['latency']]),\n  ('non-inferiority allows a small loss',\n   [[0.02, 0.01, 0.03], [['revenue', -0.015, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('readout sample 1',\n   [[0.02, 0.01, 0.04],\n    [['latency', -0.03, 0.0, 0.01, 'higher_is_better'],\n     ['crashes', -0.03, 0.020000000000000004, 0.01, 'lower_is_better']]],\n   ['blocked:latency', ['latency', 'crashes']]),\n  ('readout sample 2', [[-0.04, -0.05, -0.020000000000000004], []], ['no_ship', []]),\n  ('readout sample 3',\n   [[0.01, 0.0, 0.01],\n    [['revenue', 0.0, 0.03, 0.01, 'higher_is_better'],\n     ['latency', 0.004, 0.054000000000000006, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['latency']])],\n [('non-inferiority allows a small loss',\n   [[0.02, 0.01, 0.03], [['revenue', -0.015, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('readout sample 6',\n   [[0.01, 0.0, 0.01],\n    [['errors', -0.03, 0.0, 0.02, 'higher_is_better'],\n     ['crashes', 0.004, 0.014, 0.02, 'lower_is_better'],\n     ['revenue', -0.01, 0.019999999999999997, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['errors']]),\n  ('readout sample 7',\n   [[0.01, 0.0, 0.03],\n    [['latency', 0.004, 0.054000000000000006, 0.01, 'higher_is_better'],\n     ['errors', -0.03, 0.0, 0.02, 'higher_is_better'],\n     ['revenue', 0.004, 0.034, 0.02, 'higher_is_better']]],\n   ['inconclusive', ['errors']]),\n  ('readout sample 8',\n   [[-0.04, -0.05, -0.020000000000000004],\n    [['latency', -0.01, 0.019999999999999997, 0.01, 'lower_is_better'],\n     ['errors', 0.0, 0.03, 0.02, 'higher_is_better']]],\n   ['no_ship', ['latency']])],\n [('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('readout sample 11',\n   [[0.02, 0.01, 0.02], [['revenue', -0.01, 0.04, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('readout sample 12',\n   [[0.02, 0.01, 0.04], [['latency', -0.03, -0.019999999999999997, 0.01, 'lower_is_better']]],\n   ['ship', []]),\n  ('readout sample 13',\n   [[-0.04, -0.05, -0.04],\n    [['latency', -0.03, 0.0, 0.02, 'lower_is_better'],\n     ['revenue', 0.0, 0.03, 0.01, 'higher_is_better'],\n     ['crashes', -0.01, 0.0, 0.02, 'higher_is_better']]],\n   ['no_ship', []])],\n [('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('significant loss is no ship',\n   [[-0.02, -0.03, -0.01], [['latency', 0.0, 0.1, 0.02, 'lower_is_better']]],\n   ['no_ship', ['latency']]),\n  ('readout sample 16',\n   [[-0.01, -0.02, 0.009999999999999998],\n    [['latency', 0.004, 0.014, 0.02, 'higher_is_better'],\n     ['errors', -0.03, 0.020000000000000004, 0.01, 'lower_is_better'],\n     ['crashes', 0.004, 0.054000000000000006, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['errors', 'crashes']]),\n  ('readout sample 17',\n   [[-0.04, -0.05, -0.04], [['errors', 0.0, 0.05, 0.01, 'lower_is_better']]],\n   ['no_ship', ['errors']])],\n [('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('significant loss is no ship',\n   [[-0.02, -0.03, -0.01], [['latency', 0.0, 0.1, 0.02, 'lower_is_better']]],\n   ['no_ship', ['latency']]),\n  ('guardrail loss beyond margin',\n   [[0.02, 0.01, 0.03], [['revenue', -0.03, 0.0, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('readout sample 21',\n   [[0.01, 0.0, 0.03], [['latency', -0.02, 0.030000000000000002, 0.01, 'lower_is_better']]],\n   ['inconclusive', ['latency']])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"}},"limitations":"A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.","method":"Deterministic executable model with adversarial boundary fixtures.","provenance":{"created_by":"Failure Map","dependencies":"Python standard library","family":"w2-experiment-statistics-ship-decision-margin-strictness","generated_at":"2026-09-29T14:48:58.522147+00:00","license":"CC0-1.0","python":"3.12.14","seed":1,"split":"open-access"},"relevance":"Launch reviews combine a primary win with guardrail checks; the combination rule must be exact.","root_cause":"The check uses ci_low >= -margin.","sha256":"f6d3e7a87cec3f278949b010cb8d59948586c351a81a846db00311da50ff626e","title":"Guardrailed ship decision: A bound exactly at the margin passes · case 01","variant":1,"variant_policy":"Five numbered records share a model and may reuse boundary fixtures.","verified":true,"visibility":"public","verification":{"attempt":{"elapsed_ms":41.527,"exit_code":1,"observations":[{"actual":["blocked:latency",["latency"]],"check":"latency guardrail uses the upper bound","expected":["blocked:latency",["latency"]],"passed":true},{"actual":["ship",[]],"check":"non-inferiority allows a small loss","expected":["ship",[]],"passed":true},{"actual":["blocked:revenue",["revenue"]],"check":"bound exactly at the margin fails","expected":["blocked:revenue",["revenue"]],"passed":true},{"actual":["blocked:revenue",["revenue"]],"check":"bound just inside the margin passes","expected":["ship",[]],"passed":false},{"actual":["blocked:errors",["errors","crashes"]],"check":"first failing guardrail is named","expected":["blocked:errors",["errors","crashes"]],"passed":true},{"actual":["blocked:latency",["latency","crashes"]],"check":"readout sample 1","expected":["blocked:latency",["latency","crashes"]],"passed":true},{"actual":["no_ship",[]],"check":"readout sample 2","expected":["no_ship",[]],"passed":true},{"actual":["inconclusive",["latency"]],"check":"readout sample 3","expected":["inconclusive",["latency"]],"passed":true}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"latency guardrail uses the upper bound\", \"actual\": [\"blocked:latency\", [\"latency\"]], \"expected\": [\"blocked:latency\", [\"latency\"]], \"passed\": true}, {\"check\": \"non-inferiority allows a small loss\", \"actual\": [\"ship\", []], \"expected\": [\"ship\", []], \"passed\": true}, {\"check\": \"bound exactly at the margin fails\", \"actual\": [\"blocked:revenue\", [\"revenue\"]], \"expected\": [\"blocked:revenue\", [\"revenue\"]], \"passed\": true}, {\"check\": \"bound just inside the margin passes\", \"actual\": [\"blocked:revenue\", [\"revenue\"]], \"expected\": [\"ship\", []], \"passed\": false}, {\"check\": \"first failing guardrail is named\", \"actual\": [\"blocked:errors\", [\"errors\", \"crashes\"]], \"expected\": [\"blocked:errors\", [\"errors\", \"crashes\"]], \"passed\": true}, {\"check\": \"readout sample 1\", \"actual\": [\"blocked:latency\", [\"latency\", \"crashes\"]], \"expected\": [\"blocked:latency\", [\"latency\", \"crashes\"]], \"passed\": true}, {\"check\": \"readout sample 2\", \"actual\": [\"no_ship\", []], \"expected\": [\"no_ship\", []], \"passed\": true}, {\"check\": \"readout sample 3\", \"actual\": [\"inconclusive\", [\"latency\"]], \"expected\": [\"inconclusive\", [\"latency\"]], \"passed\": true}], \"passed\": false}\n"},"broken":{"elapsed_ms":39.831,"exit_code":1,"observations":[{"actual":["blocked:latency",["latency"]],"check":"latency guardrail uses the upper bound","expected":["blocked:latency",["latency"]],"passed":true},{"actual":["ship",[]],"check":"non-inferiority allows a small loss","expected":["ship",[]],"passed":true},{"actual":["ship",[]],"check":"bound exactly at the margin fails","expected":["blocked:revenue",["revenue"]],"passed":false},{"actual":["ship",[]],"check":"bound just inside the margin passes","expected":["ship",[]],"passed":true},{"actual":["blocked:errors",["errors","crashes"]],"check":"first failing guardrail is named","expected":["blocked:errors",["errors","crashes"]],"passed":true},{"actual":["blocked:latency",["latency","crashes"]],"check":"readout sample 1","expected":["blocked:latency",["latency","crashes"]],"passed":true},{"actual":["no_ship",[]],"check":"readout sample 2","expected":["no_ship",[]],"passed":true},{"actual":["inconclusive",["latency"]],"check":"readout sample 3","expected":["inconclusive",["latency"]],"passed":true}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"latency guardrail uses the upper bound\", \"actual\": [\"blocked:latency\", [\"latency\"]], \"expected\": [\"blocked:latency\", [\"latency\"]], \"passed\": true}, {\"check\": \"non-inferiority allows a small loss\", \"actual\": [\"ship\", []], \"expected\": [\"ship\", []], \"passed\": true}, {\"check\": \"bound exactly at the margin fails\", \"actual\": [\"ship\", []], \"expected\": [\"blocked:revenue\", [\"revenue\"]], \"passed\": false}, {\"check\": \"bound just inside the margin passes\", \"actual\": [\"ship\", []], \"expected\": [\"ship\", []], \"passed\": true}, {\"check\": \"first failing guardrail is named\", \"actual\": [\"blocked:errors\", [\"errors\", \"crashes\"]], \"expected\": [\"blocked:errors\", [\"errors\", \"crashes\"]], \"passed\": true}, {\"check\": \"readout sample 1\", \"actual\": [\"blocked:latency\", [\"latency\", \"crashes\"]], \"expected\": [\"blocked:latency\", [\"latency\", \"crashes\"]], \"passed\": true}, {\"check\": \"readout sample 2\", \"actual\": [\"no_ship\", []], \"expected\": [\"no_ship\", []], \"passed\": true}, {\"check\": \"readout sample 3\", \"actual\": [\"inconclusive\", [\"latency\"]], \"expected\": [\"inconclusive\", [\"latency\"]], \"passed\": true}], \"passed\": false}\n"}},"member_only":{"stages":["fixed"],"fields":["implementations.fixed","verification.fixed","harness","repair"],"note":"The verified repair, its recorded checks, the repair description, and the scoring harness are available to members."}}