{"abstract":"Higher-is-better guardrails demand a gain larger than the margin.","category":"Experiment statistics","checks":8,"contract":"primary is [lift, ci_low, ci_high]. Each guardrail [name, ci_low, ci_high, margin, direction] passes non-inferiority when higher_is_better and ci_low > -margin, or lower_is_better and ci_high < margin. Decision: no_ship if primary ci_high < 0; else if primary ci_low > 0 then \"blocked:<first failing guardrail in input order>\" or ship; else inconclusive. Return [decision, failing names in input order].","evaluation_group":"w2-experiment-statistics-ship-decision","failed_approach":"Requiring ci_low > 0 turns non-inferiority into superiority.","family":"w2-experiment-statistics-ship-decision-margin-sign","id":"FA-74626","implementations":{"attempt":{"sha256":"964f344fc0e18c7715e49b57a0f7ae5e74cea4ab82d7a43677b6c3c2701a34bc","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(primary, guardrails):\n    failing = []\n    for name, lo, hi, margin, direction in guardrails:\n        if direction == 'higher_is_better':\n            ok = lo > 0\n        else:\n            ok = hi < margin\n        if not ok:\n            failing.append(name)\n    if primary[2] < 0:\n        return ['no_ship', failing]\n    if primary[1] > 0:\n        if failing:\n            return ['blocked:' + failing[0], failing]\n        return ['ship', failing]\n    return ['inconclusive', failing]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('latency guardrail uses the upper bound',\n   [[0.02, 0.01, 0.03], [['latency', -0.01, 0.025, 0.02, 'lower_is_better']]],\n   ['blocked:latency', ['latency']]),\n  ('non-inferiority allows a small loss',\n   [[0.02, 0.01, 0.03], [['revenue', -0.015, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('readout sample 1',\n   [[0.02, 0.01, 0.04],\n    [['latency', -0.03, 0.0, 0.01, 'higher_is_better'],\n     ['crashes', -0.03, 0.020000000000000004, 0.01, 'lower_is_better']]],\n   ['blocked:latency', ['latency', 'crashes']]),\n  ('readout sample 2', [[-0.04, -0.05, -0.020000000000000004], []], ['no_ship', []]),\n  ('readout sample 3',\n   [[0.01, 0.0, 0.01],\n    [['revenue', 0.0, 0.03, 0.01, 'higher_is_better'],\n     ['latency', 0.004, 0.054000000000000006, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['latency']])],\n [('non-inferiority allows a small loss',\n   [[0.02, 0.01, 0.03], [['revenue', -0.015, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('readout sample 6',\n   [[0.01, 0.0, 0.01],\n    [['errors', -0.03, 0.0, 0.02, 'higher_is_better'],\n     ['crashes', 0.004, 0.014, 0.02, 'lower_is_better'],\n     ['revenue', -0.01, 0.019999999999999997, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['errors']]),\n  ('readout sample 11',\n   [[0.02, 0.01, 0.02], [['revenue', -0.01, 0.04, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('readout sample 13',\n   [[-0.04, -0.05, -0.04],\n    [['latency', -0.03, 0.0, 0.02, 'lower_is_better'],\n     ['revenue', 0.0, 0.03, 0.01, 'higher_is_better'],\n     ['crashes', -0.01, 0.0, 0.02, 'higher_is_better']]],\n   ['no_ship', []])],\n [('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('readout sample 11',\n   [[0.02, 0.01, 0.02], [['revenue', -0.01, 0.04, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('readout sample 28',\n   [[0.02, 0.01, 0.04],\n    [['revenue', 0.004, 0.054000000000000006, 0.02, 'higher_is_better'],\n     ['crashes', 0.0, 0.01, 0.02, 'higher_is_better'],\n     ['latency', -0.03, 0.0, 0.02, 'higher_is_better']]],\n   ['blocked:latency', ['latency']]),\n  ('readout sample 57',\n   [[0.02, 0.01, 0.02], [['crashes', -0.01, 0.019999999999999997, 0.02, 'higher_is_better']]],\n   ['ship', []])],\n [('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('significant loss is no ship',\n   [[-0.02, -0.03, -0.01], [['latency', 0.0, 0.1, 0.02, 'lower_is_better']]],\n   ['no_ship', ['latency']]),\n  ('readout sample 9',\n   [[0.02, 0.01, 0.02],\n    [['revenue', 0.0, 0.05, 0.01, 'higher_is_better'],\n     ['errors', 0.0, 0.03, 0.01, 'lower_is_better'],\n     ['latency', 0.0, 0.03, 0.02, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'latency']]),\n  ('readout sample 16',\n   [[-0.01, -0.02, 0.009999999999999998],\n    [['latency', 0.004, 0.014, 0.02, 'higher_is_better'],\n     ['errors', -0.03, 0.020000000000000004, 0.01, 'lower_is_better'],\n     ['crashes', 0.004, 0.054000000000000006, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['errors', 'crashes']]),\n  ('readout sample 58',\n   [[0.02, 0.01, 0.02], [['latency', 0.0, 0.01, 0.01, 'higher_is_better']]],\n   ['ship', []])],\n [('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('significant loss is no ship',\n   [[-0.02, -0.03, -0.01], [['latency', 0.0, 0.1, 0.02, 'lower_is_better']]],\n   ['no_ship', ['latency']]),\n  ('guardrail loss beyond margin',\n   [[0.02, 0.01, 0.03], [['revenue', -0.03, 0.0, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('readout sample 9',\n   [[0.02, 0.01, 0.02],\n    [['revenue', 0.0, 0.05, 0.01, 'higher_is_better'],\n     ['errors', 0.0, 0.03, 0.01, 'lower_is_better'],\n     ['latency', 0.0, 0.03, 0.02, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'latency']]),\n  ('readout sample 21',\n   [[0.01, 0.0, 0.03], [['latency', -0.02, 0.030000000000000002, 0.01, 'lower_is_better']]],\n   ['inconclusive', ['latency']]),\n  ('readout sample 43',\n   [[0.01, 0.0, 0.03], [['errors', 0.0, 0.01, 0.02, 'higher_is_better']]],\n   ['inconclusive', []])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"broken":{"sha256":"c942fd632c4b8abb33162f5a18f8099ec4f60ddfe48e9be7576cbf9327f4d9a5","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(primary, guardrails):\n    failing = []\n    for name, lo, hi, margin, direction in guardrails:\n        if direction == 'higher_is_better':\n            ok = lo > margin\n        else:\n            ok = hi < margin\n        if not ok:\n            failing.append(name)\n    if primary[2] < 0:\n        return ['no_ship', failing]\n    if primary[1] > 0:\n        if failing:\n            return ['blocked:' + failing[0], failing]\n        return ['ship', failing]\n    return ['inconclusive', failing]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('latency guardrail uses the upper bound',\n   [[0.02, 0.01, 0.03], [['latency', -0.01, 0.025, 0.02, 'lower_is_better']]],\n   ['blocked:latency', ['latency']]),\n  ('non-inferiority allows a small loss',\n   [[0.02, 0.01, 0.03], [['revenue', -0.015, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('readout sample 1',\n   [[0.02, 0.01, 0.04],\n    [['latency', -0.03, 0.0, 0.01, 'higher_is_better'],\n     ['crashes', -0.03, 0.020000000000000004, 0.01, 'lower_is_better']]],\n   ['blocked:latency', ['latency', 'crashes']]),\n  ('readout sample 2', [[-0.04, -0.05, -0.020000000000000004], []], ['no_ship', []]),\n  ('readout sample 3',\n   [[0.01, 0.0, 0.01],\n    [['revenue', 0.0, 0.03, 0.01, 'higher_is_better'],\n     ['latency', 0.004, 0.054000000000000006, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['latency']])],\n [('non-inferiority allows a small loss',\n   [[0.02, 0.01, 0.03], [['revenue', -0.015, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('readout sample 6',\n   [[0.01, 0.0, 0.01],\n    [['errors', -0.03, 0.0, 0.02, 'higher_is_better'],\n     ['crashes', 0.004, 0.014, 0.02, 'lower_is_better'],\n     ['revenue', -0.01, 0.019999999999999997, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['errors']]),\n  ('readout sample 11',\n   [[0.02, 0.01, 0.02], [['revenue', -0.01, 0.04, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('readout sample 13',\n   [[-0.04, -0.05, -0.04],\n    [['latency', -0.03, 0.0, 0.02, 'lower_is_better'],\n     ['revenue', 0.0, 0.03, 0.01, 'higher_is_better'],\n     ['crashes', -0.01, 0.0, 0.02, 'higher_is_better']]],\n   ['no_ship', []])],\n [('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('readout sample 11',\n   [[0.02, 0.01, 0.02], [['revenue', -0.01, 0.04, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('readout sample 28',\n   [[0.02, 0.01, 0.04],\n    [['revenue', 0.004, 0.054000000000000006, 0.02, 'higher_is_better'],\n     ['crashes', 0.0, 0.01, 0.02, 'higher_is_better'],\n     ['latency', -0.03, 0.0, 0.02, 'higher_is_better']]],\n   ['blocked:latency', ['latency']]),\n  ('readout sample 57',\n   [[0.02, 0.01, 0.02], [['crashes', -0.01, 0.019999999999999997, 0.02, 'higher_is_better']]],\n   ['ship', []])],\n [('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('significant loss is no ship',\n   [[-0.02, -0.03, -0.01], [['latency', 0.0, 0.1, 0.02, 'lower_is_better']]],\n   ['no_ship', ['latency']]),\n  ('readout sample 9',\n   [[0.02, 0.01, 0.02],\n    [['revenue', 0.0, 0.05, 0.01, 'higher_is_better'],\n     ['errors', 0.0, 0.03, 0.01, 'lower_is_better'],\n     ['latency', 0.0, 0.03, 0.02, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'latency']]),\n  ('readout sample 16',\n   [[-0.01, -0.02, 0.009999999999999998],\n    [['latency', 0.004, 0.014, 0.02, 'higher_is_better'],\n     ['errors', -0.03, 0.020000000000000004, 0.01, 'lower_is_better'],\n     ['crashes', 0.004, 0.054000000000000006, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['errors', 'crashes']]),\n  ('readout sample 58',\n   [[0.02, 0.01, 0.02], [['latency', 0.0, 0.01, 0.01, 'higher_is_better']]],\n   ['ship', []])],\n [('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('significant loss is no ship',\n   [[-0.02, -0.03, -0.01], [['latency', 0.0, 0.1, 0.02, 'lower_is_better']]],\n   ['no_ship', ['latency']]),\n  ('guardrail loss beyond margin',\n   [[0.02, 0.01, 0.03], [['revenue', -0.03, 0.0, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('readout sample 9',\n   [[0.02, 0.01, 0.02],\n    [['revenue', 0.0, 0.05, 0.01, 'higher_is_better'],\n     ['errors', 0.0, 0.03, 0.01, 'lower_is_better'],\n     ['latency', 0.0, 0.03, 0.02, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'latency']]),\n  ('readout sample 21',\n   [[0.01, 0.0, 0.03], [['latency', -0.02, 0.030000000000000002, 0.01, 'lower_is_better']]],\n   ['inconclusive', ['latency']]),\n  ('readout sample 43',\n   [[0.01, 0.0, 0.03], [['errors', 0.0, 0.01, 0.02, 'higher_is_better']]],\n   ['inconclusive', []])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"fixed":{"sha256":"af7f4941080df85c20ecead8923a476ab7f30a3a100fdbc66b1a086d5caa7e5c","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(primary, guardrails):\n    failing = []\n    for name, lo, hi, margin, direction in guardrails:\n        if direction == 'higher_is_better':\n            ok = lo > -margin\n        else:\n            ok = hi < margin\n        if not ok:\n            failing.append(name)\n    if primary[2] < 0:\n        return ['no_ship', failing]\n    if primary[1] > 0:\n        if failing:\n            return ['blocked:' + failing[0], failing]\n        return ['ship', failing]\n    return ['inconclusive', failing]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('latency guardrail uses the upper bound',\n   [[0.02, 0.01, 0.03], [['latency', -0.01, 0.025, 0.02, 'lower_is_better']]],\n   ['blocked:latency', ['latency']]),\n  ('non-inferiority allows a small loss',\n   [[0.02, 0.01, 0.03], [['revenue', -0.015, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('readout sample 1',\n   [[0.02, 0.01, 0.04],\n    [['latency', -0.03, 0.0, 0.01, 'higher_is_better'],\n     ['crashes', -0.03, 0.020000000000000004, 0.01, 'lower_is_better']]],\n   ['blocked:latency', ['latency', 'crashes']]),\n  ('readout sample 2', [[-0.04, -0.05, -0.020000000000000004], []], ['no_ship', []]),\n  ('readout sample 3',\n   [[0.01, 0.0, 0.01],\n    [['revenue', 0.0, 0.03, 0.01, 'higher_is_better'],\n     ['latency', 0.004, 0.054000000000000006, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['latency']])],\n [('non-inferiority allows a small loss',\n   [[0.02, 0.01, 0.03], [['revenue', -0.015, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('readout sample 6',\n   [[0.01, 0.0, 0.01],\n    [['errors', -0.03, 0.0, 0.02, 'higher_is_better'],\n     ['crashes', 0.004, 0.014, 0.02, 'lower_is_better'],\n     ['revenue', -0.01, 0.019999999999999997, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['errors']]),\n  ('readout sample 11',\n   [[0.02, 0.01, 0.02], [['revenue', -0.01, 0.04, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('readout sample 13',\n   [[-0.04, -0.05, -0.04],\n    [['latency', -0.03, 0.0, 0.02, 'lower_is_better'],\n     ['revenue', 0.0, 0.03, 0.01, 'higher_is_better'],\n     ['crashes', -0.01, 0.0, 0.02, 'higher_is_better']]],\n   ['no_ship', []])],\n [('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('readout sample 11',\n   [[0.02, 0.01, 0.02], [['revenue', -0.01, 0.04, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('readout sample 28',\n   [[0.02, 0.01, 0.04],\n    [['revenue', 0.004, 0.054000000000000006, 0.02, 'higher_is_better'],\n     ['crashes', 0.0, 0.01, 0.02, 'higher_is_better'],\n     ['latency', -0.03, 0.0, 0.02, 'higher_is_better']]],\n   ['blocked:latency', ['latency']]),\n  ('readout sample 57',\n   [[0.02, 0.01, 0.02], [['crashes', -0.01, 0.019999999999999997, 0.02, 'higher_is_better']]],\n   ['ship', []])],\n [('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('significant loss is no ship',\n   [[-0.02, -0.03, -0.01], [['latency', 0.0, 0.1, 0.02, 'lower_is_better']]],\n   ['no_ship', ['latency']]),\n  ('readout sample 9',\n   [[0.02, 0.01, 0.02],\n    [['revenue', 0.0, 0.05, 0.01, 'higher_is_better'],\n     ['errors', 0.0, 0.03, 0.01, 'lower_is_better'],\n     ['latency', 0.0, 0.03, 0.02, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'latency']]),\n  ('readout sample 16',\n   [[-0.01, -0.02, 0.009999999999999998],\n    [['latency', 0.004, 0.014, 0.02, 'higher_is_better'],\n     ['errors', -0.03, 0.020000000000000004, 0.01, 'lower_is_better'],\n     ['crashes', 0.004, 0.054000000000000006, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['errors', 'crashes']]),\n  ('readout sample 58',\n   [[0.02, 0.01, 0.02], [['latency', 0.0, 0.01, 0.01, 'higher_is_better']]],\n   ['ship', []])],\n [('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('significant loss is no ship',\n   [[-0.02, -0.03, -0.01], [['latency', 0.0, 0.1, 0.02, 'lower_is_better']]],\n   ['no_ship', ['latency']]),\n  ('guardrail loss beyond margin',\n   [[0.02, 0.01, 0.03], [['revenue', -0.03, 0.0, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('readout sample 9',\n   [[0.02, 0.01, 0.02],\n    [['revenue', 0.0, 0.05, 0.01, 'higher_is_better'],\n     ['errors', 0.0, 0.03, 0.01, 'lower_is_better'],\n     ['latency', 0.0, 0.03, 0.02, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'latency']]),\n  ('readout sample 21',\n   [[0.01, 0.0, 0.03], [['latency', -0.02, 0.030000000000000002, 0.01, 'lower_is_better']]],\n   ['inconclusive', ['latency']]),\n  ('readout sample 43',\n   [[0.01, 0.0, 0.03], [['errors', 0.0, 0.01, 0.02, 'higher_is_better']]],\n   ['inconclusive', []])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"}},"limitations":"A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.","method":"Deterministic executable model with adversarial boundary fixtures.","provenance":{"created_by":"Failure Map","dependencies":"Python standard library","family":"w2-experiment-statistics-ship-decision-margin-sign","generated_at":"2026-09-29T14:48:58.522385+00:00","license":"CC0-1.0","python":"3.12.14","seed":1,"split":"open-access"},"relevance":"Launch reviews combine a primary win with guardrail checks; the combination rule must be exact.","repair":"Allow losses down to -margin.","root_cause":"The check is ci_low > margin instead of ci_low > -margin.","sha256":"2429ba8a1d5911b51deb25fcce32e4a58483830e21439ccf86b78a7aedb6f9db","title":"Guardrailed ship decision: Non-inferiority margin has the wrong sign · case 01","variant":1,"variant_policy":"Five numbered records share a model and may reuse boundary fixtures.","verification":{"attempt":{"elapsed_ms":40.486,"exit_code":1,"observations":[{"actual":["blocked:latency",["latency"]],"check":"latency guardrail uses the upper bound","expected":["blocked:latency",["latency"]],"passed":true},{"actual":["blocked:revenue",["revenue"]],"check":"non-inferiority allows a small loss","expected":["ship",[]],"passed":false},{"actual":["blocked:revenue",["revenue"]],"check":"bound exactly at the margin fails","expected":["blocked:revenue",["revenue"]],"passed":true},{"actual":["blocked:revenue",["revenue"]],"check":"bound just inside the margin passes","expected":["ship",[]],"passed":false},{"actual":["blocked:errors",["errors","crashes"]],"check":"first failing guardrail is named","expected":["blocked:errors",["errors","crashes"]],"passed":true},{"actual":["blocked:latency",["latency","crashes"]],"check":"readout sample 1","expected":["blocked:latency",["latency","crashes"]],"passed":true},{"actual":["no_ship",[]],"check":"readout sample 2","expected":["no_ship",[]],"passed":true},{"actual":["inconclusive",["revenue","latency"]],"check":"readout sample 3","expected":["inconclusive",["latency"]],"passed":false}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"latency guardrail uses the upper bound\", \"actual\": [\"blocked:latency\", [\"latency\"]], \"expected\": [\"blocked:latency\", [\"latency\"]], \"passed\": true}, {\"check\": \"non-inferiority allows a small loss\", \"actual\": [\"blocked:revenue\", [\"revenue\"]], \"expected\": [\"ship\", []], \"passed\": false}, {\"check\": \"bound exactly at the margin fails\", \"actual\": [\"blocked:revenue\", [\"revenue\"]], \"expected\": [\"blocked:revenue\", [\"revenue\"]], \"passed\": true}, {\"check\": \"bound just inside the margin passes\", \"actual\": [\"blocked:revenue\", [\"revenue\"]], \"expected\": [\"ship\", []], \"passed\": false}, {\"check\": \"first failing guardrail is named\", \"actual\": [\"blocked:errors\", [\"errors\", \"crashes\"]], \"expected\": [\"blocked:errors\", [\"errors\", \"crashes\"]], \"passed\": true}, {\"check\": \"readout sample 1\", \"actual\": [\"blocked:latency\", [\"latency\", \"crashes\"]], \"expected\": [\"blocked:latency\", [\"latency\", \"crashes\"]], \"passed\": true}, {\"check\": \"readout sample 2\", \"actual\": [\"no_ship\", []], \"expected\": [\"no_ship\", []], \"passed\": true}, {\"check\": \"readout sample 3\", \"actual\": [\"inconclusive\", [\"revenue\", \"latency\"]], \"expected\": [\"inconclusive\", [\"latency\"]], \"passed\": false}], \"passed\": false}\n"},"broken":{"elapsed_ms":41.584,"exit_code":1,"observations":[{"actual":["blocked:latency",["latency"]],"check":"latency guardrail uses the upper bound","expected":["blocked:latency",["latency"]],"passed":true},{"actual":["blocked:revenue",["revenue"]],"check":"non-inferiority allows a small loss","expected":["ship",[]],"passed":false},{"actual":["blocked:revenue",["revenue"]],"check":"bound exactly at the margin fails","expected":["blocked:revenue",["revenue"]],"passed":true},{"actual":["blocked:revenue",["revenue"]],"check":"bound just inside the margin passes","expected":["ship",[]],"passed":false},{"actual":["blocked:errors",["errors","crashes"]],"check":"first failing guardrail is named","expected":["blocked:errors",["errors","crashes"]],"passed":true},{"actual":["blocked:latency",["latency","crashes"]],"check":"readout sample 1","expected":["blocked:latency",["latency","crashes"]],"passed":true},{"actual":["no_ship",[]],"check":"readout sample 2","expected":["no_ship",[]],"passed":true},{"actual":["inconclusive",["revenue","latency"]],"check":"readout sample 3","expected":["inconclusive",["latency"]],"passed":false}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"latency guardrail uses the upper bound\", \"actual\": [\"blocked:latency\", [\"latency\"]], \"expected\": [\"blocked:latency\", [\"latency\"]], \"passed\": true}, {\"check\": \"non-inferiority allows a small loss\", \"actual\": [\"blocked:revenue\", [\"revenue\"]], \"expected\": [\"ship\", []], \"passed\": false}, {\"check\": \"bound exactly at the margin fails\", \"actual\": [\"blocked:revenue\", [\"revenue\"]], \"expected\": [\"blocked:revenue\", [\"revenue\"]], \"passed\": true}, {\"check\": \"bound just inside the margin passes\", \"actual\": [\"blocked:revenue\", [\"revenue\"]], \"expected\": [\"ship\", []], \"passed\": false}, {\"check\": \"first failing guardrail is named\", \"actual\": [\"blocked:errors\", [\"errors\", \"crashes\"]], \"expected\": [\"blocked:errors\", [\"errors\", \"crashes\"]], \"passed\": true}, {\"check\": \"readout sample 1\", \"actual\": [\"blocked:latency\", [\"latency\", \"crashes\"]], \"expected\": [\"blocked:latency\", [\"latency\", \"crashes\"]], \"passed\": true}, {\"check\": \"readout sample 2\", \"actual\": [\"no_ship\", []], \"expected\": [\"no_ship\", []], \"passed\": true}, {\"check\": \"readout sample 3\", \"actual\": [\"inconclusive\", [\"revenue\", \"latency\"]], \"expected\": [\"inconclusive\", [\"latency\"]], \"passed\": false}], \"passed\": false}\n"},"fixed":{"elapsed_ms":40.17,"exit_code":0,"observations":[{"actual":["blocked:latency",["latency"]],"check":"latency guardrail uses the upper bound","expected":["blocked:latency",["latency"]],"passed":true},{"actual":["ship",[]],"check":"non-inferiority allows a small loss","expected":["ship",[]],"passed":true},{"actual":["blocked:revenue",["revenue"]],"check":"bound exactly at the margin fails","expected":["blocked:revenue",["revenue"]],"passed":true},{"actual":["ship",[]],"check":"bound just inside the margin passes","expected":["ship",[]],"passed":true},{"actual":["blocked:errors",["errors","crashes"]],"check":"first failing guardrail is named","expected":["blocked:errors",["errors","crashes"]],"passed":true},{"actual":["blocked:latency",["latency","crashes"]],"check":"readout sample 1","expected":["blocked:latency",["latency","crashes"]],"passed":true},{"actual":["no_ship",[]],"check":"readout sample 2","expected":["no_ship",[]],"passed":true},{"actual":["inconclusive",["latency"]],"check":"readout sample 3","expected":["inconclusive",["latency"]],"passed":true}],"passed":true,"stderr":"","stdout":"{\"observations\": [{\"check\": \"latency guardrail uses the upper bound\", \"actual\": [\"blocked:latency\", [\"latency\"]], \"expected\": [\"blocked:latency\", [\"latency\"]], \"passed\": true}, {\"check\": \"non-inferiority allows a small loss\", \"actual\": [\"ship\", []], \"expected\": [\"ship\", []], \"passed\": true}, {\"check\": \"bound exactly at the margin fails\", \"actual\": [\"blocked:revenue\", [\"revenue\"]], \"expected\": [\"blocked:revenue\", [\"revenue\"]], \"passed\": true}, {\"check\": \"bound just inside the margin passes\", \"actual\": [\"ship\", []], \"expected\": [\"ship\", []], \"passed\": true}, {\"check\": \"first failing guardrail is named\", \"actual\": [\"blocked:errors\", [\"errors\", \"crashes\"]], \"expected\": [\"blocked:errors\", [\"errors\", \"crashes\"]], \"passed\": true}, {\"check\": \"readout sample 1\", \"actual\": [\"blocked:latency\", [\"latency\", \"crashes\"]], \"expected\": [\"blocked:latency\", [\"latency\", \"crashes\"]], \"passed\": true}, {\"check\": \"readout sample 2\", \"actual\": [\"no_ship\", []], \"expected\": [\"no_ship\", []], \"passed\": true}, {\"check\": \"readout sample 3\", \"actual\": [\"inconclusive\", [\"latency\"]], \"expected\": [\"inconclusive\", [\"latency\"]], \"passed\": true}], \"passed\": true}\n"}},"verified":true,"visibility":"public"}