FA-74596 / Experiment statistics / Open access

Bayesian conversion comparison: The sum is normalised by the treatment posterior · case 01

Probabilities can exceed one or fall far below the truth.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The last term subtracts lnB(alpha_B, beta_B) instead of lnB(alpha_A, beta_A).

VERIFIED REPAIR

Normalise each term by the control posterior beta function.

Unsuccessful approach: Mixing alpha_A with beta_B in the normaliser is still the wrong beta function.

Case contract

With an integer Beta(prior_a, prior_b) prior, arm posteriors are Beta(prior_a + s, prior_b + n - s). P(B > A) uses the exact integer-parameter sum over i < alpha_B of exp(lnB(alpha_A + i, beta_A + beta_B) - ln(beta_B + i) - lnB(1 + i, beta_B) - lnB(alpha_A, beta_A)). Return [posterior mean A, posterior mean B, P(B > A)] rounded to 6.

Why this case matters

Bayesian dashboards report a "chance to beat control" that product teams act on directly.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(a_succ, a_n, b_succ, b_n, prior_a, prior_b):
    aa, ba = prior_a + a_succ, prior_b + a_n - a_succ
    ab, bb = prior_a + b_succ, prior_b + b_n - b_succ
    def lbeta(x, y):
        return math.lgamma(x) + math.lgamma(y) - math.lgamma(x + y)
    total = 0.0
    for i in range(ab):
        total += math.exp(lbeta(aa + i, ba + bb) - math.log(bb + i) - lbeta(1 + i, bb) - lbeta(ab, bb))
    return [round(aa / (aa + ba), 6), round(ab / (ab + bb), 6), round(total, 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 1', [7, 20, 5, 10, 3, 5], [0.357143, 0.444444, 0.724072]),
  ('posterior sample 2', [4, 40, 10, 10, 1, 1], [0.119048, 0.916667, 1.0])],
 [('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 3', [8, 10, 23, 40, 1, 5], [0.5625, 0.521739, 0.384139]),
  ('posterior sample 6', [13, 20, 0, 10, 2, 2], [0.625, 0.142857, 0.000906]),
  ('posterior sample 7', [2, 5, 2, 5, 2, 1], [0.5, 0.5, 0.5])],
 [('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 11', [5, 5, 30, 40, 1, 1], [0.857143, 0.738095, 0.1814]),
  ('posterior sample 12', [16, 20, 9, 20, 2, 1], [0.782609, 0.478261, 0.013376]),
  ('posterior sample 13', [7, 10, 2, 5, 3, 1], [0.714286, 0.555556, 0.212934])],
 [('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 16', [7, 10, 1, 20, 1, 2], [0.615385, 0.086957, 0.000212]),
  ('posterior sample 18', [8, 40, 14, 20, 1, 2], [0.209302, 0.652174, 0.999848]),
  ('posterior sample 19', [12, 40, 7, 40, 1, 1], [0.309524, 0.190476, 0.098949])],
 [('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 21', [6, 10, 16, 40, 1, 1], [0.583333, 0.404762, 0.132122]),
  ('posterior sample 25', [10, 10, 5, 5, 2, 1], [0.923077, 0.875, 0.368421]),
  ('posterior sample 28', [0, 5, 4, 5, 3, 1], [0.333333, 0.777778, 0.97972])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
uniform prior small counts	`[0.333333, 0.583333, 1.576417]`	`[0.333333, 0.583333, 0.90081]`	Failed
informative prior shifts means	`[0.166667, 0.25, 3.180451]`	`[0.166667, 0.25, 0.706767]`	Failed
equal data gives one half	`[0.416667, 0.416667, 0.5]`	`[0.416667, 0.416667, 0.5]`	Passed
treatment clearly worse	`[0.727273, 0.181818, 5e-06]`	`[0.727273, 0.181818, 6.3e-05]`	Failed
zero successes in treatment	`[0.25, 0.083333, 0.002381]`	`[0.25, 0.083333, 0.107143]`	Failed
unequal sample sizes	`[0.139535, 0.307692, 5.7e-05]`	`[0.139535, 0.307692, 0.902022]`	Failed
posterior sample 1	`[0.357143, 0.444444, 0.001669]`	`[0.357143, 0.444444, 0.724072]`	Failed
posterior sample 2	`[0.119048, 0.916667, 3e-06]`	`[0.119048, 0.916667, 1.0]`	Failed

SHA-256 / 0938038dc045b7b6e80fc9dac20479478ee7f09ca0a118ceef4eb26ed9c2b40d

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(a_succ, a_n, b_succ, b_n, prior_a, prior_b):
    aa, ba = prior_a + a_succ, prior_b + a_n - a_succ
    ab, bb = prior_a + b_succ, prior_b + b_n - b_succ
    def lbeta(x, y):
        return math.lgamma(x) + math.lgamma(y) - math.lgamma(x + y)
    total = 0.0
    for i in range(ab):
        total += math.exp(lbeta(aa + i, ba + bb) - math.log(bb + i) - lbeta(1 + i, bb) - lbeta(aa, bb))
    return [round(aa / (aa + ba), 6), round(ab / (ab + bb), 6), round(total, 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 1', [7, 20, 5, 10, 3, 5], [0.357143, 0.444444, 0.724072]),
  ('posterior sample 2', [4, 40, 10, 10, 1, 1], [0.119048, 0.916667, 1.0])],
 [('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 3', [8, 10, 23, 40, 1, 5], [0.5625, 0.521739, 0.384139]),
  ('posterior sample 6', [13, 20, 0, 10, 2, 2], [0.625, 0.142857, 0.000906]),
  ('posterior sample 7', [2, 5, 2, 5, 2, 1], [0.5, 0.5, 0.5])],
 [('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 11', [5, 5, 30, 40, 1, 1], [0.857143, 0.738095, 0.1814]),
  ('posterior sample 12', [16, 20, 9, 20, 2, 1], [0.782609, 0.478261, 0.013376]),
  ('posterior sample 13', [7, 10, 2, 5, 3, 1], [0.714286, 0.555556, 0.212934])],
 [('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 16', [7, 10, 1, 20, 1, 2], [0.615385, 0.086957, 0.000212]),
  ('posterior sample 18', [8, 40, 14, 20, 1, 2], [0.209302, 0.652174, 0.999848]),
  ('posterior sample 19', [12, 40, 7, 40, 1, 1], [0.309524, 0.190476, 0.098949])],
 [('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 21', [6, 10, 16, 40, 1, 1], [0.583333, 0.404762, 0.132122]),
  ('posterior sample 25', [10, 10, 5, 5, 2, 1], [0.923077, 0.875, 0.368421]),
  ('posterior sample 28', [0, 5, 4, 5, 3, 1], [0.333333, 0.777778, 0.97972])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
uniform prior small counts	`[0.333333, 0.583333, 0.191081]`	`[0.333333, 0.583333, 0.90081]`	Failed
informative prior shifts means	`[0.166667, 0.25, 0.578264]`	`[0.166667, 0.25, 0.706767]`	Failed
equal data gives one half	`[0.416667, 0.416667, 0.5]`	`[0.416667, 0.416667, 0.5]`	Passed
treatment clearly worse	`[0.727273, 0.181818, 3.638359]`	`[0.727273, 0.181818, 6.3e-05]`	Failed
zero successes in treatment	`[0.25, 0.083333, 0.185714]`	`[0.25, 0.083333, 0.107143]`	Failed
unequal sample sizes	`[0.139535, 0.307692, 0.000516]`	`[0.139535, 0.307692, 0.902022]`	Failed
posterior sample 1	`[0.357143, 0.444444, 0.007929]`	`[0.357143, 0.444444, 0.724072]`	Failed
posterior sample 2	`[0.119048, 0.916667, 1e-06]`	`[0.119048, 0.916667, 1.0]`	Failed

SHA-256 / 8f1f314be111bcec8a365c3ea13461b12d4b8754f13d858289711154edb17971

3 / The verified repair

Exit 0

"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(a_succ, a_n, b_succ, b_n, prior_a, prior_b):
    aa, ba = prior_a + a_succ, prior_b + a_n - a_succ
    ab, bb = prior_a + b_succ, prior_b + b_n - b_succ
    def lbeta(x, y):
        return math.lgamma(x) + math.lgamma(y) - math.lgamma(x + y)
    total = 0.0
    for i in range(ab):
        total += math.exp(lbeta(aa + i, ba + bb) - math.log(bb + i) - lbeta(1 + i, bb) - lbeta(aa, ba))
    return [round(aa / (aa + ba), 6), round(ab / (ab + bb), 6), round(total, 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 1', [7, 20, 5, 10, 3, 5], [0.357143, 0.444444, 0.724072]),
  ('posterior sample 2', [4, 40, 10, 10, 1, 1], [0.119048, 0.916667, 1.0])],
 [('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 3', [8, 10, 23, 40, 1, 5], [0.5625, 0.521739, 0.384139]),
  ('posterior sample 6', [13, 20, 0, 10, 2, 2], [0.625, 0.142857, 0.000906]),
  ('posterior sample 7', [2, 5, 2, 5, 2, 1], [0.5, 0.5, 0.5])],
 [('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 11', [5, 5, 30, 40, 1, 1], [0.857143, 0.738095, 0.1814]),
  ('posterior sample 12', [16, 20, 9, 20, 2, 1], [0.782609, 0.478261, 0.013376]),
  ('posterior sample 13', [7, 10, 2, 5, 3, 1], [0.714286, 0.555556, 0.212934])],
 [('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('treatment clearly worse', [15, 20, 3, 20, 1, 1], [0.727273, 0.181818, 6.3e-05]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 16', [7, 10, 1, 20, 1, 2], [0.615385, 0.086957, 0.000212]),
  ('posterior sample 18', [8, 40, 14, 20, 1, 2], [0.209302, 0.652174, 0.999848]),
  ('posterior sample 19', [12, 40, 7, 40, 1, 1], [0.309524, 0.190476, 0.098949])],
 [('uniform prior small counts', [3, 10, 6, 10, 1, 1], [0.333333, 0.583333, 0.90081]),
  ('informative prior shifts means', [0, 5, 1, 5, 2, 5], [0.166667, 0.25, 0.706767]),
  ('equal data gives one half', [4, 10, 4, 10, 1, 1], [0.416667, 0.416667, 0.5]),
  ('zero successes in treatment', [2, 10, 0, 10, 1, 1], [0.25, 0.083333, 0.107143]),
  ('unequal sample sizes', [5, 40, 3, 10, 1, 2], [0.139535, 0.307692, 0.902022]),
  ('posterior sample 21', [6, 10, 16, 40, 1, 1], [0.583333, 0.404762, 0.132122]),
  ('posterior sample 25', [10, 10, 5, 5, 2, 1], [0.923077, 0.875, 0.368421]),
  ('posterior sample 28', [0, 5, 4, 5, 3, 1], [0.333333, 0.777778, 0.97972])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
uniform prior small counts	`[0.333333, 0.583333, 0.90081]`	`[0.333333, 0.583333, 0.90081]`	Passed
informative prior shifts means	`[0.166667, 0.25, 0.706767]`	`[0.166667, 0.25, 0.706767]`	Passed
equal data gives one half	`[0.416667, 0.416667, 0.5]`	`[0.416667, 0.416667, 0.5]`	Passed
treatment clearly worse	`[0.727273, 0.181818, 6.3e-05]`	`[0.727273, 0.181818, 6.3e-05]`	Passed
zero successes in treatment	`[0.25, 0.083333, 0.107143]`	`[0.25, 0.083333, 0.107143]`	Passed
unequal sample sizes	`[0.139535, 0.307692, 0.902022]`	`[0.139535, 0.307692, 0.902022]`	Passed
posterior sample 1	`[0.357143, 0.444444, 0.724072]`	`[0.357143, 0.444444, 0.724072]`	Passed
posterior sample 2	`[0.119048, 0.916667, 1.0]`	`[0.119048, 0.916667, 1.0]`	Passed

SHA-256 / 1bd5852be89ee07433721684af5ede4a59413f84ba80ac439c7b6fcafee20bb9

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:58.282518+00:00.

Case digest / 1e658ee5fbded2075edd6d48651eb0e3206aa90f769d48c2f3fc1445c3ce1eae