FA-74831 / Experiment statistics / Open access

Percent change log-ratio interval: Standard errors are scaled by the control mean only · case 01

Intervals are wrong whenever treatment and control means differ.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The log-scale variance is (se_t^2 + se_c^2) / mean_c^2.

VERIFIED REPAIR

Scale each standard error by its own arm mean before combining.

Unsuccessful approach: Adding relative standard errors before squaring overstates the variance.

Case contract

L = ln(mean_t / mean_c) with standard error sqrt((se_t / mean_t)^2 + (se_c / mean_c)^2). Report the percent change 100 (e^L - 1) and the interval 100 (e^(L -/+ z se) - 1), each rounded to 4. Nonpositive means -> None.

Why this case matters

Log-ratio intervals keep percent-change bounds asymmetric and positive-definite.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(mean_c, se_c, mean_t, se_t, z):
    if mean_c <= 0 or mean_t <= 0:
        return None
    L = math.log(mean_t / mean_c)
    se = math.sqrt((se_t ** 2 + se_c ** 2) / mean_c ** 2)
    return [round(100 * (math.exp(L) - 1), 4), round(100 * (math.exp(L - z * se) - 1), 4), round(100 * (math.exp(L + z * se) - 1), 4)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 1', [10.0, 0.2, 10.0, 2.0, 1.645], [0.0, -28.1537, 39.186]),
  ('summary sample 2', [10.0, 1.0, 1.8, 0.3, 1.645], [-82.0, -86.9258, -75.2184]),
  ('summary sample 3', [2.0, 0.5, 1.8, 0.5, 1.96], [-10.0, -56.7354, 87.2201]),
  ('summary sample 4', [50.0, 0.2, 11.0, 2.0, 1.96], [-78.0, -84.5965, -68.5785])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 5', [2.0, 1.0, 10.0, 0.3, 1.645], [400.0, 119.3413, 1039.7763]),
  ('summary sample 6', [2.0, 0.2, 55.0, 2.0, 1.96], [2650.0, 2132.3268, 3287.7209]),
  ('summary sample 7', [50.0, 1.0, 10.0, 0.3, 1.645], [-80.0, -81.1517, -78.7779]),
  ('summary sample 8', [2.0, 0.2, 55.0, 2.0, 1.96], [2650.0, 2132.3268, 3287.7209])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 11', [2.0, 0.5, 10.0, 0.5, 1.645], [400.0, 228.7224, 660.52]),
  ('summary sample 12', [2.0, 1.0, 10.0, 0.5, 1.645], [400.0, 118.7667, 1042.7699]),
  ('summary sample 13', [10.0, 1.0, 55.0, 0.3, 1.645], [450.0, 366.4607, 548.5005]),
  ('summary sample 14', [50.0, 0.2, 11.0, 0.3, 1.645], [-78.0, -78.9753, -76.9795])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 16', [50.0, 0.2, 10.0, 0.3, 1.645], [-80.0, -80.9714, -78.9791]),
  ('summary sample 17', [50.0, 0.2, 10.0, 0.5, 1.96], [-80.0, -81.8727, -77.9338]),
  ('summary sample 19', [2.0, 1.0, 10.0, 0.3, 1.96], [400.0, 87.3251, 1234.5781]),
  ('summary sample 21', [50.0, 0.2, 1.8, 0.3, 1.645], [-96.4, -97.2635, -95.2641])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 21', [50.0, 0.2, 1.8, 0.3, 1.645], [-96.4, -97.2635, -95.2641]),
  ('summary sample 22', [10.0, 0.5, 55.0, 2.0, 1.96], [450.0, 387.2326, 520.8534]),
  ('summary sample 26', [50.0, 0.2, 55.0, 0.5, 1.96], [10.0, 7.8794, 12.1623]),
  ('summary sample 28', [10.0, 0.5, 1.8, 0.3, 1.96], [-82.0, -87.2016, -74.6844])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
ten percent increase	`[10.0, 2.4947, 18.0548]`	`[10.0, 2.9448, 17.5387]`	Failed
decrease	`[-10.0, -49.6895, 61.0001]`	`[-10.0, -51.3304, 66.4285]`	Failed
no change	`[0.0, -8.3922, 9.161]`	`[0.0, -8.3922, 9.161]`	Passed
zero control mean	`None`	`None`	Passed
summary sample 1	`[0.0, -28.1537, 39.186]`	`[0.0, -28.1537, 39.186]`	Passed
summary sample 2	`[-82.0, -84.8405, -78.6273]`	`[-82.0, -86.9258, -75.2184]`	Failed
summary sample 3	`[-10.0, -54.9918, 79.9671]`	`[-10.0, -56.7354, 87.2201]`	Failed
summary sample 4	`[-78.0, -79.6669, -76.1965]`	`[-78.0, -84.5965, -68.5785]`	Failed

SHA-256 / 9abf33d7c40649ea1c53f39144b08d3aa0b57f3f0158aa7bd69061f14089d564

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(mean_c, se_c, mean_t, se_t, z):
    if mean_c <= 0 or mean_t <= 0:
        return None
    L = math.log(mean_t / mean_c)
    se = math.sqrt((se_t / mean_t + se_c / mean_c) ** 2)
    return [round(100 * (math.exp(L) - 1), 4), round(100 * (math.exp(L - z * se) - 1), 4), round(100 * (math.exp(L + z * se) - 1), 4)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 1', [10.0, 0.2, 10.0, 2.0, 1.645], [0.0, -28.1537, 39.186]),
  ('summary sample 2', [10.0, 1.0, 1.8, 0.3, 1.645], [-82.0, -86.9258, -75.2184]),
  ('summary sample 3', [2.0, 0.5, 1.8, 0.5, 1.96], [-10.0, -56.7354, 87.2201]),
  ('summary sample 4', [50.0, 0.2, 11.0, 2.0, 1.96], [-78.0, -84.5965, -68.5785])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 5', [2.0, 1.0, 10.0, 0.3, 1.645], [400.0, 119.3413, 1039.7763]),
  ('summary sample 6', [2.0, 0.2, 55.0, 2.0, 1.96], [2650.0, 2132.3268, 3287.7209]),
  ('summary sample 7', [50.0, 1.0, 10.0, 0.3, 1.645], [-80.0, -81.1517, -78.7779]),
  ('summary sample 8', [2.0, 0.2, 55.0, 2.0, 1.96], [2650.0, 2132.3268, 3287.7209])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 11', [2.0, 0.5, 10.0, 0.5, 1.645], [400.0, 228.7224, 660.52]),
  ('summary sample 12', [2.0, 1.0, 10.0, 0.5, 1.645], [400.0, 118.7667, 1042.7699]),
  ('summary sample 13', [10.0, 1.0, 55.0, 0.3, 1.645], [450.0, 366.4607, 548.5005]),
  ('summary sample 14', [50.0, 0.2, 11.0, 0.3, 1.645], [-78.0, -78.9753, -76.9795])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 16', [50.0, 0.2, 10.0, 0.3, 1.645], [-80.0, -80.9714, -78.9791]),
  ('summary sample 17', [50.0, 0.2, 10.0, 0.5, 1.96], [-80.0, -81.8727, -77.9338]),
  ('summary sample 19', [2.0, 1.0, 10.0, 0.3, 1.96], [400.0, 87.3251, 1234.5781]),
  ('summary sample 21', [50.0, 0.2, 1.8, 0.3, 1.645], [-96.4, -97.2635, -95.2641])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 21', [50.0, 0.2, 1.8, 0.3, 1.645], [-96.4, -97.2635, -95.2641]),
  ('summary sample 22', [10.0, 0.5, 55.0, 2.0, 1.96], [450.0, 387.2326, 520.8534]),
  ('summary sample 26', [50.0, 0.2, 55.0, 0.5, 1.96], [10.0, 7.8794, 12.1623]),
  ('summary sample 28', [10.0, 0.5, 1.8, 0.3, 1.96], [-82.0, -87.2016, -74.6844])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
ten percent increase	`[10.0, 0.2659, 20.6791]`	`[10.0, 2.9448, 17.5387]`	Failed
decrease	`[-10.0, -62.2262, 114.4345]`	`[-10.0, -51.3304, 66.4285]`	Failed
no change	`[0.0, -11.0948, 12.4794]`	`[0.0, -8.3922, 9.161]`	Failed
zero control mean	`None`	`None`	Passed
summary sample 1	`[0.0, -30.3648, 43.6055]`	`[0.0, -28.1537, 39.186]`	Failed
summary sample 2	`[-82.0, -88.3919, -72.0885]`	`[-82.0, -86.9258, -75.2184]`	Failed
summary sample 3	`[-10.0, -68.0119, 153.2188]`	`[-10.0, -56.7354, 87.2201]`	Failed
summary sample 4	`[-78.0, -84.7155, -68.3339]`	`[-78.0, -84.5965, -68.5785]`	Failed

SHA-256 / 8fec8ed7ae6f935a58b85adecd4ccdc3c8c31d039e4ec90b732c403298a6187d

3 / The verified repair

Exit 0

"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(mean_c, se_c, mean_t, se_t, z):
    if mean_c <= 0 or mean_t <= 0:
        return None
    L = math.log(mean_t / mean_c)
    se = math.sqrt((se_t / mean_t) ** 2 + (se_c / mean_c) ** 2)
    return [round(100 * (math.exp(L) - 1), 4), round(100 * (math.exp(L - z * se) - 1), 4), round(100 * (math.exp(L + z * se) - 1), 4)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 1', [10.0, 0.2, 10.0, 2.0, 1.645], [0.0, -28.1537, 39.186]),
  ('summary sample 2', [10.0, 1.0, 1.8, 0.3, 1.645], [-82.0, -86.9258, -75.2184]),
  ('summary sample 3', [2.0, 0.5, 1.8, 0.5, 1.96], [-10.0, -56.7354, 87.2201]),
  ('summary sample 4', [50.0, 0.2, 11.0, 2.0, 1.96], [-78.0, -84.5965, -68.5785])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 5', [2.0, 1.0, 10.0, 0.3, 1.645], [400.0, 119.3413, 1039.7763]),
  ('summary sample 6', [2.0, 0.2, 55.0, 2.0, 1.96], [2650.0, 2132.3268, 3287.7209]),
  ('summary sample 7', [50.0, 1.0, 10.0, 0.3, 1.645], [-80.0, -81.1517, -78.7779]),
  ('summary sample 8', [2.0, 0.2, 55.0, 2.0, 1.96], [2650.0, 2132.3268, 3287.7209])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 11', [2.0, 0.5, 10.0, 0.5, 1.645], [400.0, 228.7224, 660.52]),
  ('summary sample 12', [2.0, 1.0, 10.0, 0.5, 1.645], [400.0, 118.7667, 1042.7699]),
  ('summary sample 13', [10.0, 1.0, 55.0, 0.3, 1.645], [450.0, 366.4607, 548.5005]),
  ('summary sample 14', [50.0, 0.2, 11.0, 0.3, 1.645], [-78.0, -78.9753, -76.9795])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 16', [50.0, 0.2, 10.0, 0.3, 1.645], [-80.0, -80.9714, -78.9791]),
  ('summary sample 17', [50.0, 0.2, 10.0, 0.5, 1.96], [-80.0, -81.8727, -77.9338]),
  ('summary sample 19', [2.0, 1.0, 10.0, 0.3, 1.96], [400.0, 87.3251, 1234.5781]),
  ('summary sample 21', [50.0, 0.2, 1.8, 0.3, 1.645], [-96.4, -97.2635, -95.2641])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 21', [50.0, 0.2, 1.8, 0.3, 1.645], [-96.4, -97.2635, -95.2641]),
  ('summary sample 22', [10.0, 0.5, 55.0, 2.0, 1.96], [450.0, 387.2326, 520.8534]),
  ('summary sample 26', [50.0, 0.2, 55.0, 0.5, 1.96], [10.0, 7.8794, 12.1623]),
  ('summary sample 28', [10.0, 0.5, 1.8, 0.3, 1.96], [-82.0, -87.2016, -74.6844])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
ten percent increase	`[10.0, 2.9448, 17.5387]`	`[10.0, 2.9448, 17.5387]`	Passed
decrease	`[-10.0, -51.3304, 66.4285]`	`[-10.0, -51.3304, 66.4285]`	Passed
no change	`[0.0, -8.3922, 9.161]`	`[0.0, -8.3922, 9.161]`	Passed
zero control mean	`None`	`None`	Passed
summary sample 1	`[0.0, -28.1537, 39.186]`	`[0.0, -28.1537, 39.186]`	Passed
summary sample 2	`[-82.0, -86.9258, -75.2184]`	`[-82.0, -86.9258, -75.2184]`	Passed
summary sample 3	`[-10.0, -56.7354, 87.2201]`	`[-10.0, -56.7354, 87.2201]`	Passed
summary sample 4	`[-78.0, -84.5965, -68.5785]`	`[-78.0, -84.5965, -68.5785]`	Passed

SHA-256 / 0accc0eb5c94ea8e1625ccb55708c20ae4742bbeb95f63d75fecd37a0abaca7e

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:49:00.317436+00:00.

Case digest / 77b0b0d6746960702f71432e9833b07714c18cebd476b438ddff864f1bfa1519