{"abstract":"Intervals are too narrow whenever the lift is large.","category":"Experiment statistics","checks":8,"contract":"lift = mean_t / mean_c - 1. With variances of the means v_t = var_t/n_t and v_c = var_c/n_c, the delta-method variance is v_t/mean_c^2 + mean_t^2 v_c / mean_c^4 and the interval is lift +/- z * sqrt(variance). mean_c = 0 or nonpositive n -> None. Return [lift, low, high] rounded to 6 places.","evaluation_group":"w2-experiment-statistics-relative-lift-ci","failed_approach":"Using mean_t * v_c / mean_c^3 applies only one power of the ratio.","family":"w2-experiment-statistics-relative-lift-ci-control-variance-term","id":"FA-74411","implementations":{"attempt":{"sha256":"cc23813fb38d72d902ac9c72ef43ee6dcd2798fdafb58fc78c67165811c248f1","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\nimport math\nN = 1\nobservations = []\ndef solve(mean_c, var_c, n_c, mean_t, var_t, n_t, z):\n    if mean_c == 0 or n_c <= 0 or n_t <= 0:\n        return None\n    lift = mean_t / mean_c - 1\n    se_t = var_t / n_t\n    se_c = var_c / n_c\n    v = se_t / mean_c ** 2 + mean_t * se_c / mean_c ** 3\n    half = z * math.sqrt(v)\n    return [round(lift, 6), round(lift - half, 6), round(lift + half, 6)]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('ten percent lift', [10.0, 4.0, 400, 11.0, 4.0, 400, 1.96], [0.1, 0.070863, 0.129137]),\n  ('treatment below control', [2.0, 1.0, 100, 1.5, 1.0, 100, 1.96], [-0.25, -0.3725, -0.1275]),\n  ('equal means still carry control noise',\n   [4.0, 9.0, 1000, 4.0, 1.0, 1000, 1.645],\n   [0.0, -0.041125, 0.041125]),\n  ('small means', [0.5, 0.25, 250, 0.6, 0.3, 250, 1.96], [0.2, -0.001413, 0.401413]),\n  ('zero control mean', [0.0, 1.0, 10, 1.0, 1.0, 10, 1.96], None),\n  ('summary statistic sample 1', [4.0, 4.0, 1000, 0.45, 16.0, 250, 1.96], [-0.8875, -1.01151, -0.76349]),\n  ('summary statistic sample 2', [10.0, 9.0, 400, 3.0, 1.0, 250, 1.645], [-0.7, -0.712769, -0.687231]),\n  ('summary statistic sample 3', [2.0, 1.0, 400, 4.0, 1.0, 250, 1.645], [1.0, 0.90268, 1.09732])],\n [('ten percent lift', [10.0, 4.0, 400, 11.0, 4.0, 400, 1.96], [0.1, 0.070863, 0.129137]),\n  ('treatment below control', [2.0, 1.0, 100, 1.5, 1.0, 100, 1.96], [-0.25, -0.3725, -0.1275]),\n  ('equal means still carry control noise',\n   [4.0, 9.0, 1000, 4.0, 1.0, 1000, 1.645],\n   [0.0, -0.041125, 0.041125]),\n  ('small means', [0.5, 0.25, 250, 0.6, 0.3, 250, 1.96], [0.2, -0.001413, 0.401413]),\n  ('zero control mean', [0.0, 1.0, 10, 1.0, 1.0, 10, 1.96], None),\n  ('summary statistic sample 5', [10.0, 4.0, 400, 4.0, 4.0, 1000, 1.645], [-0.6, -0.61231, -0.58769]),\n  ('summary statistic sample 6', [10.0, 9.0, 1000, 2.2, 4.0, 1000, 1.645], [-0.78, -0.790956, -0.769044]),\n  ('summary statistic sample 7', [2.0, 9.0, 400, 10.5, 1.0, 100, 1.645], [4.25, 3.59708, 4.90292])],\n [('ten percent lift', [10.0, 4.0, 400, 11.0, 4.0, 400, 1.96], [0.1, 0.070863, 0.129137]),\n  ('treatment below control', [2.0, 1.0, 100, 1.5, 1.0, 100, 1.96], [-0.25, -0.3725, -0.1275]),\n  ('equal means still carry control noise',\n   [4.0, 9.0, 1000, 4.0, 1.0, 1000, 1.645],\n   [0.0, -0.041125, 0.041125]),\n  ('small means', [0.5, 0.25, 250, 0.6, 0.3, 250, 1.96], [0.2, -0.001413, 0.401413]),\n  ('zero control mean', [0.0, 1.0, 10, 1.0, 1.0, 10, 1.96], None),\n  ('summary statistic sample 11', [4.0, 4.0, 400, 10.5, 16.0, 1000, 1.96], [1.625, 1.48222, 1.76778]),\n  ('summary statistic sample 12', [10.0, 9.0, 400, 0.45, 1.0, 250, 1.96], [-0.955, -0.967467, -0.942533]),\n  ('summary statistic sample 13', [0.5, 1.0, 1000, 10.5, 4.0, 100, 1.645], [20.0, 17.718248, 22.281752])],\n [('ten percent lift', [10.0, 4.0, 400, 11.0, 4.0, 400, 1.96], [0.1, 0.070863, 0.129137]),\n  ('treatment below control', [2.0, 1.0, 100, 1.5, 1.0, 100, 1.96], [-0.25, -0.3725, -0.1275]),\n  ('equal means still carry control noise',\n   [4.0, 9.0, 1000, 4.0, 1.0, 1000, 1.645],\n   [0.0, -0.041125, 0.041125]),\n  ('small means', [0.5, 0.25, 250, 0.6, 0.3, 250, 1.96], [0.2, -0.001413, 0.401413]),\n  ('zero control mean', [0.0, 1.0, 10, 1.0, 1.0, 10, 1.96], None),\n  ('summary statistic sample 16', [4.0, 9.0, 100, 4.0, 4.0, 250, 1.645], [0.0, -0.133893, 0.133893]),\n  ('summary statistic sample 17', [4.0, 1.0, 100, 4.0, 16.0, 1000, 1.645], [0.0, -0.066312, 0.066312]),\n  ('summary statistic sample 22', [0.5, 4.0, 1000, 10.5, 4.0, 1000, 1.96], [20.0, 14.787726, 25.212274])],\n [('ten percent lift', [10.0, 4.0, 400, 11.0, 4.0, 400, 1.96], [0.1, 0.070863, 0.129137]),\n  ('treatment below control', [2.0, 1.0, 100, 1.5, 1.0, 100, 1.96], [-0.25, -0.3725, -0.1275]),\n  ('equal means still carry control noise',\n   [4.0, 9.0, 1000, 4.0, 1.0, 1000, 1.645],\n   [0.0, -0.041125, 0.041125]),\n  ('small means', [0.5, 0.25, 250, 0.6, 0.3, 250, 1.96], [0.2, -0.001413, 0.401413]),\n  ('zero control mean', [0.0, 1.0, 10, 1.0, 1.0, 10, 1.96], None),\n  ('summary statistic sample 21', [0.5, 1.0, 400, 0.45, 1.0, 100, 1.96], [-0.1, -0.529862, 0.329862]),\n  ('summary statistic sample 22', [0.5, 4.0, 1000, 10.5, 4.0, 1000, 1.96], [20.0, 14.787726, 25.212274]),\n  ('summary statistic sample 31', [0.5, 1.0, 100, 4.0, 16.0, 1000, 1.96], [7.0, 3.825042, 10.174958])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"broken":{"sha256":"729b876b47db506fbe44efd6263f87cda73feece42d483f86fd64ab05fd3aa29","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\nimport math\nN = 1\nobservations = []\ndef solve(mean_c, var_c, n_c, mean_t, var_t, n_t, z):\n    if mean_c == 0 or n_c <= 0 or n_t <= 0:\n        return None\n    lift = mean_t / mean_c - 1\n    se_t = var_t / n_t\n    se_c = var_c / n_c\n    v = se_t / mean_c ** 2 + se_c / mean_c ** 2\n    half = z * math.sqrt(v)\n    return [round(lift, 6), round(lift - half, 6), round(lift + half, 6)]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('ten percent lift', [10.0, 4.0, 400, 11.0, 4.0, 400, 1.96], [0.1, 0.070863, 0.129137]),\n  ('treatment below control', [2.0, 1.0, 100, 1.5, 1.0, 100, 1.96], [-0.25, -0.3725, -0.1275]),\n  ('equal means still carry control noise',\n   [4.0, 9.0, 1000, 4.0, 1.0, 1000, 1.645],\n   [0.0, -0.041125, 0.041125]),\n  ('small means', [0.5, 0.25, 250, 0.6, 0.3, 250, 1.96], [0.2, -0.001413, 0.401413]),\n  ('zero control mean', [0.0, 1.0, 10, 1.0, 1.0, 10, 1.96], None),\n  ('summary statistic sample 1', [4.0, 4.0, 1000, 0.45, 16.0, 250, 1.96], [-0.8875, -1.01151, -0.76349]),\n  ('summary statistic sample 2', [10.0, 9.0, 400, 3.0, 1.0, 250, 1.645], [-0.7, -0.712769, -0.687231]),\n  ('summary statistic sample 3', [2.0, 1.0, 400, 4.0, 1.0, 250, 1.645], [1.0, 0.90268, 1.09732])],\n [('ten percent lift', [10.0, 4.0, 400, 11.0, 4.0, 400, 1.96], [0.1, 0.070863, 0.129137]),\n  ('treatment below control', [2.0, 1.0, 100, 1.5, 1.0, 100, 1.96], [-0.25, -0.3725, -0.1275]),\n  ('equal means still carry control noise',\n   [4.0, 9.0, 1000, 4.0, 1.0, 1000, 1.645],\n   [0.0, -0.041125, 0.041125]),\n  ('small means', [0.5, 0.25, 250, 0.6, 0.3, 250, 1.96], [0.2, -0.001413, 0.401413]),\n  ('zero control mean', [0.0, 1.0, 10, 1.0, 1.0, 10, 1.96], None),\n  ('summary statistic sample 5', [10.0, 4.0, 400, 4.0, 4.0, 1000, 1.645], [-0.6, -0.61231, -0.58769]),\n  ('summary statistic sample 6', [10.0, 9.0, 1000, 2.2, 4.0, 1000, 1.645], [-0.78, -0.790956, -0.769044]),\n  ('summary statistic sample 7', [2.0, 9.0, 400, 10.5, 1.0, 100, 1.645], [4.25, 3.59708, 4.90292])],\n [('ten percent lift', [10.0, 4.0, 400, 11.0, 4.0, 400, 1.96], [0.1, 0.070863, 0.129137]),\n  ('treatment below control', [2.0, 1.0, 100, 1.5, 1.0, 100, 1.96], [-0.25, -0.3725, -0.1275]),\n  ('equal means still carry control noise',\n   [4.0, 9.0, 1000, 4.0, 1.0, 1000, 1.645],\n   [0.0, -0.041125, 0.041125]),\n  ('small means', [0.5, 0.25, 250, 0.6, 0.3, 250, 1.96], [0.2, -0.001413, 0.401413]),\n  ('zero control mean', [0.0, 1.0, 10, 1.0, 1.0, 10, 1.96], None),\n  ('summary statistic sample 11', [4.0, 4.0, 400, 10.5, 16.0, 1000, 1.96], [1.625, 1.48222, 1.76778]),\n  ('summary statistic sample 12', [10.0, 9.0, 400, 0.45, 1.0, 250, 1.96], [-0.955, -0.967467, -0.942533]),\n  ('summary statistic sample 13', [0.5, 1.0, 1000, 10.5, 4.0, 100, 1.645], [20.0, 17.718248, 22.281752])],\n [('ten percent lift', [10.0, 4.0, 400, 11.0, 4.0, 400, 1.96], [0.1, 0.070863, 0.129137]),\n  ('treatment below control', [2.0, 1.0, 100, 1.5, 1.0, 100, 1.96], [-0.25, -0.3725, -0.1275]),\n  ('equal means still carry control noise',\n   [4.0, 9.0, 1000, 4.0, 1.0, 1000, 1.645],\n   [0.0, -0.041125, 0.041125]),\n  ('small means', [0.5, 0.25, 250, 0.6, 0.3, 250, 1.96], [0.2, -0.001413, 0.401413]),\n  ('zero control mean', [0.0, 1.0, 10, 1.0, 1.0, 10, 1.96], None),\n  ('summary statistic sample 16', [4.0, 9.0, 100, 4.0, 4.0, 250, 1.645], [0.0, -0.133893, 0.133893]),\n  ('summary statistic sample 17', [4.0, 1.0, 100, 4.0, 16.0, 1000, 1.645], [0.0, -0.066312, 0.066312]),\n  ('summary statistic sample 22', [0.5, 4.0, 1000, 10.5, 4.0, 1000, 1.96], [20.0, 14.787726, 25.212274])],\n [('ten percent lift', [10.0, 4.0, 400, 11.0, 4.0, 400, 1.96], [0.1, 0.070863, 0.129137]),\n  ('treatment below control', [2.0, 1.0, 100, 1.5, 1.0, 100, 1.96], [-0.25, -0.3725, -0.1275]),\n  ('equal means still carry control noise',\n   [4.0, 9.0, 1000, 4.0, 1.0, 1000, 1.645],\n   [0.0, -0.041125, 0.041125]),\n  ('small means', [0.5, 0.25, 250, 0.6, 0.3, 250, 1.96], [0.2, -0.001413, 0.401413]),\n  ('zero control mean', [0.0, 1.0, 10, 1.0, 1.0, 10, 1.96], None),\n  ('summary statistic sample 21', [0.5, 1.0, 400, 0.45, 1.0, 100, 1.96], [-0.1, -0.529862, 0.329862]),\n  ('summary statistic sample 22', [0.5, 4.0, 1000, 10.5, 4.0, 1000, 1.96], [20.0, 14.787726, 25.212274]),\n  ('summary statistic sample 31', [0.5, 1.0, 100, 4.0, 16.0, 1000, 1.96], [7.0, 3.825042, 10.174958])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"fixed":{"sha256":"93edf1568328ee8496d5df5e297007792669b0554c2e83f592814c5b92fe7db4","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\nimport math\nN = 1\nobservations = []\ndef solve(mean_c, var_c, n_c, mean_t, var_t, n_t, z):\n    if mean_c == 0 or n_c <= 0 or n_t <= 0:\n        return None\n    lift = mean_t / mean_c - 1\n    se_t = var_t / n_t\n    se_c = var_c / n_c\n    v = se_t / mean_c ** 2 + mean_t ** 2 * se_c / mean_c ** 4\n    half = z * math.sqrt(v)\n    return [round(lift, 6), round(lift - half, 6), round(lift + half, 6)]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('ten percent lift', [10.0, 4.0, 400, 11.0, 4.0, 400, 1.96], [0.1, 0.070863, 0.129137]),\n  ('treatment below control', [2.0, 1.0, 100, 1.5, 1.0, 100, 1.96], [-0.25, -0.3725, -0.1275]),\n  ('equal means still carry control noise',\n   [4.0, 9.0, 1000, 4.0, 1.0, 1000, 1.645],\n   [0.0, -0.041125, 0.041125]),\n  ('small means', [0.5, 0.25, 250, 0.6, 0.3, 250, 1.96], [0.2, -0.001413, 0.401413]),\n  ('zero control mean', [0.0, 1.0, 10, 1.0, 1.0, 10, 1.96], None),\n  ('summary statistic sample 1', [4.0, 4.0, 1000, 0.45, 16.0, 250, 1.96], [-0.8875, -1.01151, -0.76349]),\n  ('summary statistic sample 2', [10.0, 9.0, 400, 3.0, 1.0, 250, 1.645], [-0.7, -0.712769, -0.687231]),\n  ('summary statistic sample 3', [2.0, 1.0, 400, 4.0, 1.0, 250, 1.645], [1.0, 0.90268, 1.09732])],\n [('ten percent lift', [10.0, 4.0, 400, 11.0, 4.0, 400, 1.96], [0.1, 0.070863, 0.129137]),\n  ('treatment below control', [2.0, 1.0, 100, 1.5, 1.0, 100, 1.96], [-0.25, -0.3725, -0.1275]),\n  ('equal means still carry control noise',\n   [4.0, 9.0, 1000, 4.0, 1.0, 1000, 1.645],\n   [0.0, -0.041125, 0.041125]),\n  ('small means', [0.5, 0.25, 250, 0.6, 0.3, 250, 1.96], [0.2, -0.001413, 0.401413]),\n  ('zero control mean', [0.0, 1.0, 10, 1.0, 1.0, 10, 1.96], None),\n  ('summary statistic sample 5', [10.0, 4.0, 400, 4.0, 4.0, 1000, 1.645], [-0.6, -0.61231, -0.58769]),\n  ('summary statistic sample 6', [10.0, 9.0, 1000, 2.2, 4.0, 1000, 1.645], [-0.78, -0.790956, -0.769044]),\n  ('summary statistic sample 7', [2.0, 9.0, 400, 10.5, 1.0, 100, 1.645], [4.25, 3.59708, 4.90292])],\n [('ten percent lift', [10.0, 4.0, 400, 11.0, 4.0, 400, 1.96], [0.1, 0.070863, 0.129137]),\n  ('treatment below control', [2.0, 1.0, 100, 1.5, 1.0, 100, 1.96], [-0.25, -0.3725, -0.1275]),\n  ('equal means still carry control noise',\n   [4.0, 9.0, 1000, 4.0, 1.0, 1000, 1.645],\n   [0.0, -0.041125, 0.041125]),\n  ('small means', [0.5, 0.25, 250, 0.6, 0.3, 250, 1.96], [0.2, -0.001413, 0.401413]),\n  ('zero control mean', [0.0, 1.0, 10, 1.0, 1.0, 10, 1.96], None),\n  ('summary statistic sample 11', [4.0, 4.0, 400, 10.5, 16.0, 1000, 1.96], [1.625, 1.48222, 1.76778]),\n  ('summary statistic sample 12', [10.0, 9.0, 400, 0.45, 1.0, 250, 1.96], [-0.955, -0.967467, -0.942533]),\n  ('summary statistic sample 13', [0.5, 1.0, 1000, 10.5, 4.0, 100, 1.645], [20.0, 17.718248, 22.281752])],\n [('ten percent lift', [10.0, 4.0, 400, 11.0, 4.0, 400, 1.96], [0.1, 0.070863, 0.129137]),\n  ('treatment below control', [2.0, 1.0, 100, 1.5, 1.0, 100, 1.96], [-0.25, -0.3725, -0.1275]),\n  ('equal means still carry control noise',\n   [4.0, 9.0, 1000, 4.0, 1.0, 1000, 1.645],\n   [0.0, -0.041125, 0.041125]),\n  ('small means', [0.5, 0.25, 250, 0.6, 0.3, 250, 1.96], [0.2, -0.001413, 0.401413]),\n  ('zero control mean', [0.0, 1.0, 10, 1.0, 1.0, 10, 1.96], None),\n  ('summary statistic sample 16', [4.0, 9.0, 100, 4.0, 4.0, 250, 1.645], [0.0, -0.133893, 0.133893]),\n  ('summary statistic sample 17', [4.0, 1.0, 100, 4.0, 16.0, 1000, 1.645], [0.0, -0.066312, 0.066312]),\n  ('summary statistic sample 22', [0.5, 4.0, 1000, 10.5, 4.0, 1000, 1.96], [20.0, 14.787726, 25.212274])],\n [('ten percent lift', [10.0, 4.0, 400, 11.0, 4.0, 400, 1.96], [0.1, 0.070863, 0.129137]),\n  ('treatment below control', [2.0, 1.0, 100, 1.5, 1.0, 100, 1.96], [-0.25, -0.3725, -0.1275]),\n  ('equal means still carry control noise',\n   [4.0, 9.0, 1000, 4.0, 1.0, 1000, 1.645],\n   [0.0, -0.041125, 0.041125]),\n  ('small means', [0.5, 0.25, 250, 0.6, 0.3, 250, 1.96], [0.2, -0.001413, 0.401413]),\n  ('zero control mean', [0.0, 1.0, 10, 1.0, 1.0, 10, 1.96], None),\n  ('summary statistic sample 21', [0.5, 1.0, 400, 0.45, 1.0, 100, 1.96], [-0.1, -0.529862, 0.329862]),\n  ('summary statistic sample 22', [0.5, 4.0, 1000, 10.5, 4.0, 1000, 1.96], [20.0, 14.787726, 25.212274]),\n  ('summary statistic sample 31', [0.5, 1.0, 100, 4.0, 16.0, 1000, 1.96], [7.0, 3.825042, 10.174958])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"}},"limitations":"A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.","method":"Deterministic executable model with adversarial boundary fixtures.","provenance":{"created_by":"Failure Map","dependencies":"Python standard library","family":"w2-experiment-statistics-relative-lift-ci-control-variance-term","generated_at":"2026-09-29T14:48:56.590416+00:00","license":"CC0-1.0","python":"3.12.14","seed":1,"split":"open-access"},"relevance":"Online experiment readouts drive launch decisions; a silent formula slip flips conclusions.","repair":"Use mean_t^2 * v_c / mean_c^4 for the control term.","root_cause":"The control term is v_c / mean_c^2, dropping the mean_t^2 / mean_c^2 factor.","sha256":"1cd69b91b84d450cb5611385386f4216c39aa136d8d9f49b54bd05a4fbe59e48","title":"Relative lift interval: Control noise is scaled like treatment noise · case 01","variant":1,"variant_policy":"Five numbered records share a model and may reuse boundary fixtures.","verification":{"attempt":{"elapsed_ms":38.849,"exit_code":1,"observations":[{"actual":[0.1,0.071597,0.128403],"check":"ten percent lift","expected":[0.1,0.070863,0.129137],"passed":false},{"actual":[-0.25,-0.379642,-0.120358],"check":"treatment below control","expected":[-0.25,-0.3725,-0.1275],"passed":false},{"actual":[0.0,-0.041125,0.041125],"check":"equal means still carry control noise","expected":[0.0,-0.041125,0.041125],"passed":true},{"actual":[0.2,0.00796,0.39204],"check":"small means","expected":[0.2,-0.001413,0.401413],"passed":false},{"actual":null,"check":"zero control mean","expected":null,"passed":true},{"actual":[-0.8875,-1.011896,-0.763104],"check":"summary statistic sample 1","expected":[-0.8875,-1.01151,-0.76349],"passed":false},{"actual":[-0.7,-0.717056,-0.682944],"check":"summary statistic sample 2","expected":[-0.7,-0.712769,-0.687231],"passed":false},{"actual":[1.0,0.921971,1.078029],"check":"summary statistic sample 3","expected":[1.0,0.90268,1.09732],"passed":false}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"ten percent lift\", \"actual\": [0.1, 0.071597, 0.128403], \"expected\": [0.1, 0.070863, 0.129137], \"passed\": false}, {\"check\": \"treatment below control\", \"actual\": [-0.25, -0.379642, -0.120358], \"expected\": [-0.25, -0.3725, -0.1275], \"passed\": false}, {\"check\": \"equal means still carry control noise\", \"actual\": [0.0, -0.041125, 0.041125], \"expected\": [0.0, -0.041125, 0.041125], \"passed\": true}, {\"check\": \"small means\", \"actual\": [0.2, 0.00796, 0.39204], \"expected\": [0.2, -0.001413, 0.401413], \"passed\": false}, {\"check\": \"zero control mean\", \"actual\": null, \"expected\": null, \"passed\": true}, {\"check\": \"summary statistic sample 1\", \"actual\": [-0.8875, -1.011896, -0.763104], \"expected\": [-0.8875, -1.01151, -0.76349], \"passed\": false}, {\"check\": \"summary statistic sample 2\", \"actual\": [-0.7, -0.717056, -0.682944], \"expected\": [-0.7, -0.712769, -0.687231], \"passed\": false}, {\"check\": \"summary statistic sample 3\", \"actual\": [1.0, 0.921971, 1.078029], \"expected\": [1.0, 0.90268, 1.09732], \"passed\": false}], \"passed\": false}\n"},"broken":{"elapsed_ms":39.058,"exit_code":1,"observations":[{"actual":[0.1,0.072281,0.127719],"check":"ten percent lift","expected":[0.1,0.070863,0.129137],"passed":false},{"actual":[-0.25,-0.388593,-0.111407],"check":"treatment below control","expected":[-0.25,-0.3725,-0.1275],"passed":false},{"actual":[0.0,-0.041125,0.041125],"check":"equal means still carry control noise","expected":[0.0,-0.041125,0.041125],"passed":true},{"actual":[0.2,0.016136,0.383864],"check":"small means","expected":[0.2,-0.001413,0.401413],"passed":false},{"actual":null,"check":"zero control mean","expected":null,"passed":true},{"actual":[-0.8875,-1.015276,-0.759724],"check":"summary statistic sample 1","expected":[-0.8875,-1.01151,-0.76349],"passed":false},{"actual":[-0.7,-0.726779,-0.673221],"check":"summary statistic sample 2","expected":[-0.7,-0.712769,-0.687231],"passed":false},{"actual":[1.0,0.933688,1.066312],"check":"summary statistic sample 3","expected":[1.0,0.90268,1.09732],"passed":false}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"ten percent lift\", \"actual\": [0.1, 0.072281, 0.127719], \"expected\": [0.1, 0.070863, 0.129137], \"passed\": false}, {\"check\": \"treatment below control\", \"actual\": [-0.25, -0.388593, -0.111407], \"expected\": [-0.25, -0.3725, -0.1275], \"passed\": false}, {\"check\": \"equal means still carry control noise\", \"actual\": [0.0, -0.041125, 0.041125], \"expected\": [0.0, -0.041125, 0.041125], \"passed\": true}, {\"check\": \"small means\", \"actual\": [0.2, 0.016136, 0.383864], \"expected\": [0.2, -0.001413, 0.401413], \"passed\": false}, {\"check\": \"zero control mean\", \"actual\": null, \"expected\": null, \"passed\": true}, {\"check\": \"summary statistic sample 1\", \"actual\": [-0.8875, -1.015276, -0.759724], \"expected\": [-0.8875, -1.01151, -0.76349], \"passed\": false}, {\"check\": \"summary statistic sample 2\", \"actual\": [-0.7, -0.726779, -0.673221], \"expected\": [-0.7, -0.712769, -0.687231], \"passed\": false}, {\"check\": \"summary statistic sample 3\", \"actual\": [1.0, 0.933688, 1.066312], \"expected\": [1.0, 0.90268, 1.09732], \"passed\": false}], \"passed\": false}\n"},"fixed":{"elapsed_ms":40.732,"exit_code":0,"observations":[{"actual":[0.1,0.070863,0.129137],"check":"ten percent lift","expected":[0.1,0.070863,0.129137],"passed":true},{"actual":[-0.25,-0.3725,-0.1275],"check":"treatment below control","expected":[-0.25,-0.3725,-0.1275],"passed":true},{"actual":[0.0,-0.041125,0.041125],"check":"equal means still carry control noise","expected":[0.0,-0.041125,0.041125],"passed":true},{"actual":[0.2,-0.001413,0.401413],"check":"small means","expected":[0.2,-0.001413,0.401413],"passed":true},{"actual":null,"check":"zero control mean","expected":null,"passed":true},{"actual":[-0.8875,-1.01151,-0.76349],"check":"summary statistic sample 1","expected":[-0.8875,-1.01151,-0.76349],"passed":true},{"actual":[-0.7,-0.712769,-0.687231],"check":"summary statistic sample 2","expected":[-0.7,-0.712769,-0.687231],"passed":true},{"actual":[1.0,0.90268,1.09732],"check":"summary statistic sample 3","expected":[1.0,0.90268,1.09732],"passed":true}],"passed":true,"stderr":"","stdout":"{\"observations\": [{\"check\": \"ten percent lift\", \"actual\": [0.1, 0.070863, 0.129137], \"expected\": [0.1, 0.070863, 0.129137], \"passed\": true}, {\"check\": \"treatment below control\", \"actual\": [-0.25, -0.3725, -0.1275], \"expected\": [-0.25, -0.3725, -0.1275], \"passed\": true}, {\"check\": \"equal means still carry control noise\", \"actual\": [0.0, -0.041125, 0.041125], \"expected\": [0.0, -0.041125, 0.041125], \"passed\": true}, {\"check\": \"small means\", \"actual\": [0.2, -0.001413, 0.401413], \"expected\": [0.2, -0.001413, 0.401413], \"passed\": true}, {\"check\": \"zero control mean\", \"actual\": null, \"expected\": null, \"passed\": true}, {\"check\": \"summary statistic sample 1\", \"actual\": [-0.8875, -1.01151, -0.76349], \"expected\": [-0.8875, -1.01151, -0.76349], \"passed\": true}, {\"check\": \"summary statistic sample 2\", \"actual\": [-0.7, -0.712769, -0.687231], \"expected\": [-0.7, -0.712769, -0.687231], \"passed\": true}, {\"check\": \"summary statistic sample 3\", \"actual\": [1.0, 0.90268, 1.09732], \"expected\": [1.0, 0.90268, 1.09732], \"passed\": true}], \"passed\": true}\n"}},"verified":true,"visibility":"public"}