FA-74581 / Experiment statistics / Open access

Bayesian conversion comparison: Posterior beta ignores observed successes · case 01

Posterior means are too low because every trial is counted as a failure.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

beta is prior_b + n rather than prior_b + n - successes.

VERIFIED REPAIR

Add only the failures n - s to the beta parameter.

Unsuccessful approach: Dropping the prior from beta makes zero-failure arms degenerate and ignores prior strength.

Case contract

With an integer Beta(prior_a, prior_b) prior, arm posteriors are Beta(prior_a + s, prior_b + n - s). P(B > A) uses the exact integer-parameter sum over i < alpha_B of exp(lnB(alpha_A + i, beta_A + beta_B) - ln(beta_B + i) - lnB(1 + i, beta_B) - lnB(alpha_A, beta_A)). Return [posterior mean A, posterior mean B, P(B > A)] rounded to 6.

Why this case matters

Bayesian dashboards report a "chance to beat control" that product teams act on directly.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(a_succ, a_n, b_succ, b_n, prior_a, prior_b):
    aa, ba = prior_a + a_succ, prior_b + a_n
    ab, bb = prior_a + b_succ, prior_b + b_n
    def lbeta(x, y):
        return math.lgamma(x) + math.lgamma(y) - math.lgamma(x + y)
    total = 0.0
    for i in range(ab):
        total += math.exp(lbeta(aa + i, ba + bb) - math.log(bb + i) - lbeta(1 + i, bb) - lbeta(aa, ba))
    return [round(aa / (aa + ba), 6), round(ab / (ab + bb), 6), round(total, 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 1', [7, 20, 5, 10, 3, 5], [0.357143, 0.444444, 0.724072]),
  ('posterior sample 2', [4, 40, 10, 10, 1, 1], [0.119048, 0.916667, 1.0])],
 [('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 2', [4, 40, 10, 10, 1, 1], [0.119048, 0.916667, 1.0]),
  ('posterior sample 6', [13, 20, 0, 10, 2, 2], [0.625, 0.142857, 0.000906]),
  ('posterior sample 7', [2, 5, 2, 5, 2, 1], [0.5, 0.5, 0.5])],
 [('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 9', [7, 10, 3, 5, 1, 1], [0.666667, 0.571429, 0.33872]),
  ('posterior sample 11', [5, 5, 30, 40, 1, 1], [0.857143, 0.738095, 0.1814]),
  ('posterior sample 12', [16, 20, 9, 20, 2, 1], [0.782609, 0.478261, 0.013376])],
 [('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 16', [7, 10, 1, 20, 1, 2], [0.615385, 0.086957, 0.000212]),
  ('posterior sample 17', [9, 10, 8, 10, 1, 2], [0.769231, 0.692308, 0.320203]),
  ('posterior sample 18', [8, 40, 14, 20, 1, 2], [0.209302, 0.652174, 0.999848])],
 [('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 21', [6, 10, 16, 40, 1, 1], [0.583333, 0.404762, 0.132122]),
  ('posterior sample 22', [7, 10, 18, 20, 1, 5], [0.5, 0.730769, 0.937656]),
  ('posterior sample 23', [18, 20, 0, 5, 1, 2], [0.826087, 0.125, 7.7e-05])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
uniform prior small counts	`[0.266667, 0.388889, 0.782399]`	`[0.333333, 0.583333, 0.90081]`	Failed
informative prior shifts means	`[0.166667, 0.230769, 0.670807]`	`[0.166667, 0.25, 0.706767]`	Failed
equal data gives one half	`[0.3125, 0.3125, 0.5]`	`[0.416667, 0.416667, 0.5]`	Failed
treatment clearly worse	`[0.432432, 0.16, 0.008505]`	`[0.727273, 0.181818, 6.3e-05]`	Failed
zero successes in treatment	`[0.214286, 0.083333, 0.141304]`	`[0.25, 0.083333, 0.107143]`	Failed
unequal sample sizes	`[0.125, 0.25, 0.866026]`	`[0.139535, 0.307692, 0.902022]`	Failed
posterior sample 1	`[0.285714, 0.347826, 0.68942]`	`[0.357143, 0.444444, 0.724072]`	Failed
posterior sample 2	`[0.108696, 0.5, 0.999788]`	`[0.119048, 0.916667, 1.0]`	Failed

SHA-256 / bca2aff71569193262e33db2ca3eec720b7033b5d47c923f5400d80e244a5b04

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(a_succ, a_n, b_succ, b_n, prior_a, prior_b):
    aa, ba = prior_a + a_succ, max(a_n - a_succ, 1)
    ab, bb = prior_a + b_succ, max(b_n - b_succ, 1)
    def lbeta(x, y):
        return math.lgamma(x) + math.lgamma(y) - math.lgamma(x + y)
    total = 0.0
    for i in range(ab):
        total += math.exp(lbeta(aa + i, ba + bb) - math.log(bb + i) - lbeta(1 + i, bb) - lbeta(aa, ba))
    return [round(aa / (aa + ba), 6), round(ab / (ab + bb), 6), round(total, 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 1', [7, 20, 5, 10, 3, 5], [0.357143, 0.444444, 0.724072]),
  ('posterior sample 2', [4, 40, 10, 10, 1, 1], [0.119048, 0.916667, 1.0])],
 [('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 2', [4, 40, 10, 10, 1, 1], [0.119048, 0.916667, 1.0]),
  ('posterior sample 6', [13, 20, 0, 10, 2, 2], [0.625, 0.142857, 0.000906]),
  ('posterior sample 7', [2, 5, 2, 5, 2, 1], [0.5, 0.5, 0.5])],
 [('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 9', [7, 10, 3, 5, 1, 1], [0.666667, 0.571429, 0.33872]),
  ('posterior sample 11', [5, 5, 30, 40, 1, 1], [0.857143, 0.738095, 0.1814]),
  ('posterior sample 12', [16, 20, 9, 20, 2, 1], [0.782609, 0.478261, 0.013376])],
 [('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 16', [7, 10, 1, 20, 1, 2], [0.615385, 0.086957, 0.000212]),
  ('posterior sample 17', [9, 10, 8, 10, 1, 2], [0.769231, 0.692308, 0.320203]),
  ('posterior sample 18', [8, 40, 14, 20, 1, 2], [0.209302, 0.652174, 0.999848])],
 [('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 21', [6, 10, 16, 40, 1, 1], [0.583333, 0.404762, 0.132122]),
  ('posterior sample 22', [7, 10, 18, 20, 1, 5], [0.5, 0.730769, 0.937656]),
  ('posterior sample 23', [18, 20, 0, 5, 1, 2], [0.826087, 0.125, 7.7e-05])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
uniform prior small counts	`[0.363636, 0.636364, 0.910552]`	`[0.333333, 0.583333, 0.90081]`	Failed
informative prior shifts means	`[0.285714, 0.428571, 0.727273]`	`[0.166667, 0.25, 0.706767]`	Failed
equal data gives one half	`[0.454545, 0.454545, 0.5]`	`[0.416667, 0.416667, 0.5]`	Failed
treatment clearly worse	`[0.761905, 0.190476, 4.4e-05]`	`[0.727273, 0.181818, 6.3e-05]`	Failed
zero successes in treatment	`[0.272727, 0.090909, 0.105263]`	`[0.25, 0.083333, 0.107143]`	Failed
unequal sample sizes	`[0.146341, 0.363636, 0.934789]`	`[0.139535, 0.307692, 0.902022]`	Failed
posterior sample 1	`[0.434783, 0.615385, 0.859124]`	`[0.357143, 0.444444, 0.724072]`	Failed
posterior sample 2	`[0.121951, 0.916667, 1.0]`	`[0.119048, 0.916667, 1.0]`	Failed

SHA-256 / 18c3b6a2496f2a09fa97bfdd319733f9c585b7043b1e1d29e4c72f4ff66737d9

3 / The verified repair

Exit 0

"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(a_succ, a_n, b_succ, b_n, prior_a, prior_b):
    aa, ba = prior_a + a_succ, prior_b + a_n - a_succ
    ab, bb = prior_a + b_succ, prior_b + b_n - b_succ
    def lbeta(x, y):
        return math.lgamma(x) + math.lgamma(y) - math.lgamma(x + y)
    total = 0.0
    for i in range(ab):
        total += math.exp(lbeta(aa + i, ba + bb) - math.log(bb + i) - lbeta(1 + i, bb) - lbeta(aa, ba))
    return [round(aa / (aa + ba), 6), round(ab / (ab + bb), 6), round(total, 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 1', [7, 20, 5, 10, 3, 5], [0.357143, 0.444444, 0.724072]),
  ('posterior sample 2', [4, 40, 10, 10, 1, 1], [0.119048, 0.916667, 1.0])],
 [('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 2', [4, 40, 10, 10, 1, 1], [0.119048, 0.916667, 1.0]),
  ('posterior sample 6', [13, 20, 0, 10, 2, 2], [0.625, 0.142857, 0.000906]),
  ('posterior sample 7', [2, 5, 2, 5, 2, 1], [0.5, 0.5, 0.5])],
 [('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 9', [7, 10, 3, 5, 1, 1], [0.666667, 0.571429, 0.33872]),
  ('posterior sample 11', [5, 5, 30, 40, 1, 1], [0.857143, 0.738095, 0.1814]),
  ('posterior sample 12', [16, 20, 9, 20, 2, 1], [0.782609, 0.478261, 0.013376])],
 [('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 16', [7, 10, 1, 20, 1, 2], [0.615385, 0.086957, 0.000212]),
  ('posterior sample 17', [9, 10, 8, 10, 1, 2], [0.769231, 0.692308, 0.320203]),
  ('posterior sample 18', [8, 40, 14, 20, 1, 2], [0.209302, 0.652174, 0.999848])],
 [('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 21', [6, 10, 16, 40, 1, 1], [0.583333, 0.404762, 0.132122]),
  ('posterior sample 22', [7, 10, 18, 20, 1, 5], [0.5, 0.730769, 0.937656]),
  ('posterior sample 23', [18, 20, 0, 5, 1, 2], [0.826087, 0.125, 7.7e-05])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
uniform prior small counts	`[0.333333, 0.583333, 0.90081]`	`[0.333333, 0.583333, 0.90081]`	Passed
informative prior shifts means	`[0.166667, 0.25, 0.706767]`	`[0.166667, 0.25, 0.706767]`	Passed
equal data gives one half	`[0.416667, 0.416667, 0.5]`	`[0.416667, 0.416667, 0.5]`	Passed
treatment clearly worse	`[0.727273, 0.181818, 6.3e-05]`	`[0.727273, 0.181818, 6.3e-05]`	Passed
zero successes in treatment	`[0.25, 0.083333, 0.107143]`	`[0.25, 0.083333, 0.107143]`	Passed
unequal sample sizes	`[0.139535, 0.307692, 0.902022]`	`[0.139535, 0.307692, 0.902022]`	Passed
posterior sample 1	`[0.357143, 0.444444, 0.724072]`	`[0.357143, 0.444444, 0.724072]`	Passed
posterior sample 2	`[0.119048, 0.916667, 1.0]`	`[0.119048, 0.916667, 1.0]`	Passed

SHA-256 / 4effd8f61813ac918ca6968a9095fedd2d529ddeb4dacfcdb0787d8a90fee7a9

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:58.156183+00:00.

Case digest / 88fa9bcb248b9a6692083c1c88fc57ca86c7fd84908f2fb2eaeb6da7a7e6a989