{"abstract":"Launches ship on noise whenever the lift is positive.","category":"Experiment statistics","checks":8,"contract":"primary is [lift, ci_low, ci_high]. Each guardrail [name, ci_low, ci_high, margin, direction] passes non-inferiority when higher_is_better and ci_low > -margin, or lower_is_better and ci_high < margin. Decision: no_ship if primary ci_high < 0; else if primary ci_low > 0 then \"blocked:<first failing guardrail in input order>\" or ship; else inconclusive. Return [decision, failing names in input order].","evaluation_group":"w2-experiment-statistics-ship-decision","failed_approach":"Accepting a lower bound equal to zero still ships inconclusive results.","family":"w2-experiment-statistics-ship-decision-primary-significance","id":"FA-74641","implementations":{"attempt":{"sha256":"988bcc73d1d77135efe189eaa6c38e8a33b01c611d852a951c8409e416970a15","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(primary, guardrails):\n    failing = []\n    for name, lo, hi, margin, direction in guardrails:\n        if direction == 'higher_is_better':\n            ok = lo > -margin\n        else:\n            ok = hi < margin\n        if not ok:\n            failing.append(name)\n    if primary[2] < 0:\n        return ['no_ship', failing]\n    if primary[1] >= 0:\n        if failing:\n            return ['blocked:' + failing[0], failing]\n        return ['ship', failing]\n    return ['inconclusive', failing]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('latency guardrail uses the upper bound',\n   [[0.02, 0.01, 0.03], [['latency', -0.01, 0.025, 0.02, 'lower_is_better']]],\n   ['blocked:latency', ['latency']]),\n  ('non-inferiority allows a small loss',\n   [[0.02, 0.01, 0.03], [['revenue', -0.015, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('readout sample 1',\n   [[0.02, 0.01, 0.04],\n    [['latency', -0.03, 0.0, 0.01, 'higher_is_better'],\n     ['crashes', -0.03, 0.020000000000000004, 0.01, 'lower_is_better']]],\n   ['blocked:latency', ['latency', 'crashes']])],\n [('non-inferiority allows a small loss',\n   [[0.02, 0.01, 0.03], [['revenue', -0.015, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('readout sample 25', [[0.01, 0.0, 0.01], []], ['inconclusive', []]),\n  ('readout sample 30',\n   [[0.01, 0.0, 0.01], [['latency', 0.004, 0.014, 0.02, 'lower_is_better']]],\n   ['inconclusive', []])],\n [('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('readout sample 3',\n   [[0.01, 0.0, 0.01],\n    [['revenue', 0.0, 0.03, 0.01, 'higher_is_better'],\n     ['latency', 0.004, 0.054000000000000006, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['latency']]),\n  ('readout sample 11',\n   [[0.02, 0.01, 0.02], [['revenue', -0.01, 0.04, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('readout sample 12',\n   [[0.02, 0.01, 0.04], [['latency', -0.03, -0.019999999999999997, 0.01, 'lower_is_better']]],\n   ['ship', []])],\n [('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('significant loss is no ship',\n   [[-0.02, -0.03, -0.01], [['latency', 0.0, 0.1, 0.02, 'lower_is_better']]],\n   ['no_ship', ['latency']]),\n  ('readout sample 16',\n   [[-0.01, -0.02, 0.009999999999999998],\n    [['latency', 0.004, 0.014, 0.02, 'higher_is_better'],\n     ['errors', -0.03, 0.020000000000000004, 0.01, 'lower_is_better'],\n     ['crashes', 0.004, 0.054000000000000006, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['errors', 'crashes']]),\n  ('readout sample 25', [[0.01, 0.0, 0.01], []], ['inconclusive', []]),\n  ('readout sample 32',\n   [[0.01, 0.0, 0.03],\n    [['latency', 0.0, 0.01, 0.02, 'higher_is_better'], ['revenue', 0.0, 0.01, 0.01, 'higher_is_better']]],\n   ['inconclusive', []])],\n [('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('significant loss is no ship',\n   [[-0.02, -0.03, -0.01], [['latency', 0.0, 0.1, 0.02, 'lower_is_better']]],\n   ['no_ship', ['latency']]),\n  ('guardrail loss beyond margin',\n   [[0.02, 0.01, 0.03], [['revenue', -0.03, 0.0, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('readout sample 6',\n   [[0.01, 0.0, 0.01],\n    [['errors', -0.03, 0.0, 0.02, 'higher_is_better'],\n     ['crashes', 0.004, 0.014, 0.02, 'lower_is_better'],\n     ['revenue', -0.01, 0.019999999999999997, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['errors']]),\n  ('readout sample 21',\n   [[0.01, 0.0, 0.03], [['latency', -0.02, 0.030000000000000002, 0.01, 'lower_is_better']]],\n   ['inconclusive', ['latency']]),\n  ('readout sample 22',\n   [[0.011, 0.001, 0.011], [['crashes', 0.004, 0.034, 0.01, 'higher_is_better']]],\n   ['ship', []])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"broken":{"sha256":"c7c9c3d7b3ded10f5e58fe8771c5d2c50c5ea091416578e78ba3e46e682beae4","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(primary, guardrails):\n    failing = []\n    for name, lo, hi, margin, direction in guardrails:\n        if direction == 'higher_is_better':\n            ok = lo > -margin\n        else:\n            ok = hi < margin\n        if not ok:\n            failing.append(name)\n    if primary[2] < 0:\n        return ['no_ship', failing]\n    if primary[0] > 0:\n        if failing:\n            return ['blocked:' + failing[0], failing]\n        return ['ship', failing]\n    return ['inconclusive', failing]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('latency guardrail uses the upper bound',\n   [[0.02, 0.01, 0.03], [['latency', -0.01, 0.025, 0.02, 'lower_is_better']]],\n   ['blocked:latency', ['latency']]),\n  ('non-inferiority allows a small loss',\n   [[0.02, 0.01, 0.03], [['revenue', -0.015, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('readout sample 1',\n   [[0.02, 0.01, 0.04],\n    [['latency', -0.03, 0.0, 0.01, 'higher_is_better'],\n     ['crashes', -0.03, 0.020000000000000004, 0.01, 'lower_is_better']]],\n   ['blocked:latency', ['latency', 'crashes']])],\n [('non-inferiority allows a small loss',\n   [[0.02, 0.01, 0.03], [['revenue', -0.015, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('readout sample 25', [[0.01, 0.0, 0.01], []], ['inconclusive', []]),\n  ('readout sample 30',\n   [[0.01, 0.0, 0.01], [['latency', 0.004, 0.014, 0.02, 'lower_is_better']]],\n   ['inconclusive', []])],\n [('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('readout sample 3',\n   [[0.01, 0.0, 0.01],\n    [['revenue', 0.0, 0.03, 0.01, 'higher_is_better'],\n     ['latency', 0.004, 0.054000000000000006, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['latency']]),\n  ('readout sample 11',\n   [[0.02, 0.01, 0.02], [['revenue', -0.01, 0.04, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('readout sample 12',\n   [[0.02, 0.01, 0.04], [['latency', -0.03, -0.019999999999999997, 0.01, 'lower_is_better']]],\n   ['ship', []])],\n [('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('significant loss is no ship',\n   [[-0.02, -0.03, -0.01], [['latency', 0.0, 0.1, 0.02, 'lower_is_better']]],\n   ['no_ship', ['latency']]),\n  ('readout sample 16',\n   [[-0.01, -0.02, 0.009999999999999998],\n    [['latency', 0.004, 0.014, 0.02, 'higher_is_better'],\n     ['errors', -0.03, 0.020000000000000004, 0.01, 'lower_is_better'],\n     ['crashes', 0.004, 0.054000000000000006, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['errors', 'crashes']]),\n  ('readout sample 25', [[0.01, 0.0, 0.01], []], ['inconclusive', []]),\n  ('readout sample 32',\n   [[0.01, 0.0, 0.03],\n    [['latency', 0.0, 0.01, 0.02, 'higher_is_better'], ['revenue', 0.0, 0.01, 0.01, 'higher_is_better']]],\n   ['inconclusive', []])],\n [('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('significant loss is no ship',\n   [[-0.02, -0.03, -0.01], [['latency', 0.0, 0.1, 0.02, 'lower_is_better']]],\n   ['no_ship', ['latency']]),\n  ('guardrail loss beyond margin',\n   [[0.02, 0.01, 0.03], [['revenue', -0.03, 0.0, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('readout sample 6',\n   [[0.01, 0.0, 0.01],\n    [['errors', -0.03, 0.0, 0.02, 'higher_is_better'],\n     ['crashes', 0.004, 0.014, 0.02, 'lower_is_better'],\n     ['revenue', -0.01, 0.019999999999999997, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['errors']]),\n  ('readout sample 21',\n   [[0.01, 0.0, 0.03], [['latency', -0.02, 0.030000000000000002, 0.01, 'lower_is_better']]],\n   ['inconclusive', ['latency']]),\n  ('readout sample 22',\n   [[0.011, 0.001, 0.011], [['crashes', 0.004, 0.034, 0.01, 'higher_is_better']]],\n   ['ship', []])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"fixed":{"sha256":"3a30f247ee1178fff72781d5341408c944386fd04864775bf4b24ef38927aa8b","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(primary, guardrails):\n    failing = []\n    for name, lo, hi, margin, direction in guardrails:\n        if direction == 'higher_is_better':\n            ok = lo > -margin\n        else:\n            ok = hi < margin\n        if not ok:\n            failing.append(name)\n    if primary[2] < 0:\n        return ['no_ship', failing]\n    if primary[1] > 0:\n        if failing:\n            return ['blocked:' + failing[0], failing]\n        return ['ship', failing]\n    return ['inconclusive', failing]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('latency guardrail uses the upper bound',\n   [[0.02, 0.01, 0.03], [['latency', -0.01, 0.025, 0.02, 'lower_is_better']]],\n   ['blocked:latency', ['latency']]),\n  ('non-inferiority allows a small loss',\n   [[0.02, 0.01, 0.03], [['revenue', -0.015, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('readout sample 1',\n   [[0.02, 0.01, 0.04],\n    [['latency', -0.03, 0.0, 0.01, 'higher_is_better'],\n     ['crashes', -0.03, 0.020000000000000004, 0.01, 'lower_is_better']]],\n   ['blocked:latency', ['latency', 'crashes']])],\n [('non-inferiority allows a small loss',\n   [[0.02, 0.01, 0.03], [['revenue', -0.015, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('readout sample 25', [[0.01, 0.0, 0.01], []], ['inconclusive', []]),\n  ('readout sample 30',\n   [[0.01, 0.0, 0.01], [['latency', 0.004, 0.014, 0.02, 'lower_is_better']]],\n   ['inconclusive', []])],\n [('bound exactly at the margin fails',\n   [[0.02, 0.01, 0.03], [['revenue', -0.02, 0.01, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('readout sample 3',\n   [[0.01, 0.0, 0.01],\n    [['revenue', 0.0, 0.03, 0.01, 'higher_is_better'],\n     ['latency', 0.004, 0.054000000000000006, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['latency']]),\n  ('readout sample 11',\n   [[0.02, 0.01, 0.02], [['revenue', -0.01, 0.04, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('readout sample 12',\n   [[0.02, 0.01, 0.04], [['latency', -0.03, -0.019999999999999997, 0.01, 'lower_is_better']]],\n   ['ship', []])],\n [('bound just inside the margin passes',\n   [[0.02, 0.01, 0.03], [['revenue', -0.0196, 0.01, 0.02, 'higher_is_better']]],\n   ['ship', []]),\n  ('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('significant loss is no ship',\n   [[-0.02, -0.03, -0.01], [['latency', 0.0, 0.1, 0.02, 'lower_is_better']]],\n   ['no_ship', ['latency']]),\n  ('readout sample 16',\n   [[-0.01, -0.02, 0.009999999999999998],\n    [['latency', 0.004, 0.014, 0.02, 'higher_is_better'],\n     ['errors', -0.03, 0.020000000000000004, 0.01, 'lower_is_better'],\n     ['crashes', 0.004, 0.054000000000000006, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['errors', 'crashes']]),\n  ('readout sample 25', [[0.01, 0.0, 0.01], []], ['inconclusive', []]),\n  ('readout sample 32',\n   [[0.01, 0.0, 0.03],\n    [['latency', 0.0, 0.01, 0.02, 'higher_is_better'], ['revenue', 0.0, 0.01, 0.01, 'higher_is_better']]],\n   ['inconclusive', []])],\n [('first failing guardrail is named',\n   [[0.02, 0.01, 0.03],\n    [['errors', 0.0, 0.05, 0.02, 'lower_is_better'], ['crashes', 0.0, 0.03, 0.01, 'lower_is_better']]],\n   ['blocked:errors', ['errors', 'crashes']]),\n  ('positive point estimate without significance', [[0.01, -0.005, 0.025], []], ['inconclusive', []]),\n  ('interval touching zero is inconclusive', [[0.01, 0.0, 0.02], []], ['inconclusive', []]),\n  ('significant loss is no ship',\n   [[-0.02, -0.03, -0.01], [['latency', 0.0, 0.1, 0.02, 'lower_is_better']]],\n   ['no_ship', ['latency']]),\n  ('guardrail loss beyond margin',\n   [[0.02, 0.01, 0.03], [['revenue', -0.03, 0.0, 0.02, 'higher_is_better']]],\n   ['blocked:revenue', ['revenue']]),\n  ('readout sample 6',\n   [[0.01, 0.0, 0.01],\n    [['errors', -0.03, 0.0, 0.02, 'higher_is_better'],\n     ['crashes', 0.004, 0.014, 0.02, 'lower_is_better'],\n     ['revenue', -0.01, 0.019999999999999997, 0.02, 'lower_is_better']]],\n   ['inconclusive', ['errors']]),\n  ('readout sample 21',\n   [[0.01, 0.0, 0.03], [['latency', -0.02, 0.030000000000000002, 0.01, 'lower_is_better']]],\n   ['inconclusive', ['latency']]),\n  ('readout sample 22',\n   [[0.011, 0.001, 0.011], [['crashes', 0.004, 0.034, 0.01, 'higher_is_better']]],\n   ['ship', []])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"}},"limitations":"A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.","method":"Deterministic executable model with adversarial boundary fixtures.","provenance":{"created_by":"Failure Map","dependencies":"Python standard library","family":"w2-experiment-statistics-ship-decision-primary-significance","generated_at":"2026-09-29T14:48:58.619939+00:00","license":"CC0-1.0","python":"3.12.14","seed":1,"split":"open-access"},"relevance":"Launch reviews combine a primary win with guardrail checks; the combination rule must be exact.","repair":"Require the lower confidence bound to be above zero.","root_cause":"The ship branch tests the point estimate primary[0] > 0.","sha256":"a06b880b337d946cb63a1dfbec07cbce8b031360921ce42a3e1e06edb05835d4","title":"Guardrailed ship decision: A positive point estimate counts as a win · case 01","variant":1,"variant_policy":"Five numbered records share a model and may reuse boundary fixtures.","verification":{"attempt":{"elapsed_ms":40.868,"exit_code":1,"observations":[{"actual":["blocked:latency",["latency"]],"check":"latency guardrail uses the upper bound","expected":["blocked:latency",["latency"]],"passed":true},{"actual":["ship",[]],"check":"non-inferiority allows a small loss","expected":["ship",[]],"passed":true},{"actual":["blocked:revenue",["revenue"]],"check":"bound exactly at the margin fails","expected":["blocked:revenue",["revenue"]],"passed":true},{"actual":["ship",[]],"check":"bound just inside the margin passes","expected":["ship",[]],"passed":true},{"actual":["blocked:errors",["errors","crashes"]],"check":"first failing guardrail is named","expected":["blocked:errors",["errors","crashes"]],"passed":true},{"actual":["inconclusive",[]],"check":"positive point estimate without significance","expected":["inconclusive",[]],"passed":true},{"actual":["ship",[]],"check":"interval touching zero is inconclusive","expected":["inconclusive",[]],"passed":false},{"actual":["blocked:latency",["latency","crashes"]],"check":"readout sample 1","expected":["blocked:latency",["latency","crashes"]],"passed":true}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"latency guardrail uses the upper bound\", \"actual\": [\"blocked:latency\", [\"latency\"]], \"expected\": [\"blocked:latency\", [\"latency\"]], \"passed\": true}, {\"check\": \"non-inferiority allows a small loss\", \"actual\": [\"ship\", []], \"expected\": [\"ship\", []], \"passed\": true}, {\"check\": \"bound exactly at the margin fails\", \"actual\": [\"blocked:revenue\", [\"revenue\"]], \"expected\": [\"blocked:revenue\", [\"revenue\"]], \"passed\": true}, {\"check\": \"bound just inside the margin passes\", \"actual\": [\"ship\", []], \"expected\": [\"ship\", []], \"passed\": true}, {\"check\": \"first failing guardrail is named\", \"actual\": [\"blocked:errors\", [\"errors\", \"crashes\"]], \"expected\": [\"blocked:errors\", [\"errors\", \"crashes\"]], \"passed\": true}, {\"check\": \"positive point estimate without significance\", \"actual\": [\"inconclusive\", []], \"expected\": [\"inconclusive\", []], \"passed\": true}, {\"check\": \"interval touching zero is inconclusive\", \"actual\": [\"ship\", []], \"expected\": [\"inconclusive\", []], \"passed\": false}, {\"check\": \"readout sample 1\", \"actual\": [\"blocked:latency\", [\"latency\", \"crashes\"]], \"expected\": [\"blocked:latency\", [\"latency\", \"crashes\"]], \"passed\": true}], \"passed\": false}\n"},"broken":{"elapsed_ms":41.28,"exit_code":1,"observations":[{"actual":["blocked:latency",["latency"]],"check":"latency guardrail uses the upper bound","expected":["blocked:latency",["latency"]],"passed":true},{"actual":["ship",[]],"check":"non-inferiority allows a small loss","expected":["ship",[]],"passed":true},{"actual":["blocked:revenue",["revenue"]],"check":"bound exactly at the margin fails","expected":["blocked:revenue",["revenue"]],"passed":true},{"actual":["ship",[]],"check":"bound just inside the margin passes","expected":["ship",[]],"passed":true},{"actual":["blocked:errors",["errors","crashes"]],"check":"first failing guardrail is named","expected":["blocked:errors",["errors","crashes"]],"passed":true},{"actual":["ship",[]],"check":"positive point estimate without significance","expected":["inconclusive",[]],"passed":false},{"actual":["ship",[]],"check":"interval touching zero is inconclusive","expected":["inconclusive",[]],"passed":false},{"actual":["blocked:latency",["latency","crashes"]],"check":"readout sample 1","expected":["blocked:latency",["latency","crashes"]],"passed":true}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"latency guardrail uses the upper bound\", \"actual\": [\"blocked:latency\", [\"latency\"]], \"expected\": [\"blocked:latency\", [\"latency\"]], \"passed\": true}, {\"check\": \"non-inferiority allows a small loss\", \"actual\": [\"ship\", []], \"expected\": [\"ship\", []], \"passed\": true}, {\"check\": \"bound exactly at the margin fails\", \"actual\": [\"blocked:revenue\", [\"revenue\"]], \"expected\": [\"blocked:revenue\", [\"revenue\"]], \"passed\": true}, {\"check\": \"bound just inside the margin passes\", \"actual\": [\"ship\", []], \"expected\": [\"ship\", []], \"passed\": true}, {\"check\": \"first failing guardrail is named\", \"actual\": [\"blocked:errors\", [\"errors\", \"crashes\"]], \"expected\": [\"blocked:errors\", [\"errors\", \"crashes\"]], \"passed\": true}, {\"check\": \"positive point estimate without significance\", \"actual\": [\"ship\", []], \"expected\": [\"inconclusive\", []], \"passed\": false}, {\"check\": \"interval touching zero is inconclusive\", \"actual\": [\"ship\", []], \"expected\": [\"inconclusive\", []], \"passed\": false}, {\"check\": \"readout sample 1\", \"actual\": [\"blocked:latency\", [\"latency\", \"crashes\"]], \"expected\": [\"blocked:latency\", [\"latency\", \"crashes\"]], \"passed\": true}], \"passed\": false}\n"},"fixed":{"elapsed_ms":41.736,"exit_code":0,"observations":[{"actual":["blocked:latency",["latency"]],"check":"latency guardrail uses the upper bound","expected":["blocked:latency",["latency"]],"passed":true},{"actual":["ship",[]],"check":"non-inferiority allows a small loss","expected":["ship",[]],"passed":true},{"actual":["blocked:revenue",["revenue"]],"check":"bound exactly at the margin fails","expected":["blocked:revenue",["revenue"]],"passed":true},{"actual":["ship",[]],"check":"bound just inside the margin passes","expected":["ship",[]],"passed":true},{"actual":["blocked:errors",["errors","crashes"]],"check":"first failing guardrail is named","expected":["blocked:errors",["errors","crashes"]],"passed":true},{"actual":["inconclusive",[]],"check":"positive point estimate without significance","expected":["inconclusive",[]],"passed":true},{"actual":["inconclusive",[]],"check":"interval touching zero is inconclusive","expected":["inconclusive",[]],"passed":true},{"actual":["blocked:latency",["latency","crashes"]],"check":"readout sample 1","expected":["blocked:latency",["latency","crashes"]],"passed":true}],"passed":true,"stderr":"","stdout":"{\"observations\": [{\"check\": \"latency guardrail uses the upper bound\", \"actual\": [\"blocked:latency\", [\"latency\"]], \"expected\": [\"blocked:latency\", [\"latency\"]], \"passed\": true}, {\"check\": \"non-inferiority allows a small loss\", \"actual\": [\"ship\", []], \"expected\": [\"ship\", []], \"passed\": true}, {\"check\": \"bound exactly at the margin fails\", \"actual\": [\"blocked:revenue\", [\"revenue\"]], \"expected\": [\"blocked:revenue\", [\"revenue\"]], \"passed\": true}, {\"check\": \"bound just inside the margin passes\", \"actual\": [\"ship\", []], \"expected\": [\"ship\", []], \"passed\": true}, {\"check\": \"first failing guardrail is named\", \"actual\": [\"blocked:errors\", [\"errors\", \"crashes\"]], \"expected\": [\"blocked:errors\", [\"errors\", \"crashes\"]], \"passed\": true}, {\"check\": \"positive point estimate without significance\", \"actual\": [\"inconclusive\", []], \"expected\": [\"inconclusive\", []], \"passed\": true}, {\"check\": \"interval touching zero is inconclusive\", \"actual\": [\"inconclusive\", []], \"expected\": [\"inconclusive\", []], \"passed\": true}, {\"check\": \"readout sample 1\", \"actual\": [\"blocked:latency\", [\"latency\", \"crashes\"]], \"expected\": [\"blocked:latency\", [\"latency\", \"crashes\"]], \"passed\": true}], \"passed\": true}\n"}},"verified":true,"visibility":"public"}