FA-74836 / Experiment statistics / Open access

Percent change log-ratio interval: The point estimate is reported in log points · case 01

The headline change disagrees with its own interval.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The estimate is 100 L instead of 100 (e^L - 1).

VERIFIED REPAIR

Report 100 (e^L - 1).

Unsuccessful approach: Dropping the factor 100 reports a fraction beside percent bounds.

Case contract

L = ln(mean_t / mean_c) with standard error sqrt((se_t / mean_t)^2 + (se_c / mean_c)^2). Report the percent change 100 (e^L - 1) and the interval 100 (e^(L -/+ z se) - 1), each rounded to 4. Nonpositive means -> None.

Why this case matters

Log-ratio intervals keep percent-change bounds asymmetric and positive-definite.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(mean_c, se_c, mean_t, se_t, z):
    if mean_c <= 0 or mean_t <= 0:
        return None
    L = math.log(mean_t / mean_c)
    se = math.sqrt((se_t / mean_t) ** 2 + (se_c / mean_c) ** 2)
    return [round(100 * L, 4), round(100 * (math.exp(L - z * se) - 1), 4), round(100 * (math.exp(L + z * se) - 1), 4)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 1', [10.0, 0.2, 10.0, 2.0, 1.645], [0.0, -28.1537, 39.186]),
  ('summary sample 2', [10.0, 1.0, 1.8, 0.3, 1.645], [-82.0, -86.9258, -75.2184]),
  ('summary sample 3', [2.0, 0.5, 1.8, 0.5, 1.96], [-10.0, -56.7354, 87.2201]),
  ('summary sample 4', [50.0, 0.2, 11.0, 2.0, 1.96], [-78.0, -84.5965, -68.5785])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 6', [2.0, 0.2, 55.0, 2.0, 1.96], [2650.0, 2132.3268, 3287.7209]),
  ('summary sample 7', [50.0, 1.0, 10.0, 0.3, 1.645], [-80.0, -81.1517, -78.7779]),
  ('summary sample 8', [2.0, 0.2, 55.0, 2.0, 1.96], [2650.0, 2132.3268, 3287.7209]),
  ('summary sample 9', [50.0, 1.0, 11.0, 0.5, 1.645], [-78.0, -79.7257, -76.1274])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 11', [2.0, 0.5, 10.0, 0.5, 1.645], [400.0, 228.7224, 660.52]),
  ('summary sample 12', [2.0, 1.0, 10.0, 0.5, 1.645], [400.0, 118.7667, 1042.7699]),
  ('summary sample 13', [10.0, 1.0, 55.0, 0.3, 1.645], [450.0, 366.4607, 548.5005]),
  ('summary sample 14', [50.0, 0.2, 11.0, 0.3, 1.645], [-78.0, -78.9753, -76.9795])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 16', [50.0, 0.2, 10.0, 0.3, 1.645], [-80.0, -80.9714, -78.9791]),
  ('summary sample 17', [50.0, 0.2, 10.0, 0.5, 1.96], [-80.0, -81.8727, -77.9338]),
  ('summary sample 18', [2.0, 0.5, 11.0, 0.5, 1.96], [450.0, 234.2486, 805.0151]),
  ('summary sample 21', [50.0, 0.2, 1.8, 0.3, 1.645], [-96.4, -97.2635, -95.2641])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 21', [50.0, 0.2, 1.8, 0.3, 1.645], [-96.4, -97.2635, -95.2641]),
  ('summary sample 22', [10.0, 0.5, 55.0, 2.0, 1.96], [450.0, 387.2326, 520.8534]),
  ('summary sample 23', [2.0, 1.0, 10.0, 0.5, 1.96], [400.0, 86.7406, 1238.7557]),
  ('summary sample 28', [10.0, 0.5, 1.8, 0.3, 1.96], [-82.0, -87.2016, -74.6844])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
ten percent increase	`[9.531, 2.9448, 17.5387]`	`[10.0, 2.9448, 17.5387]`	Failed
decrease	`[-10.5361, -51.3304, 66.4285]`	`[-10.0, -51.3304, 66.4285]`	Failed
no change	`[0.0, -8.3922, 9.161]`	`[0.0, -8.3922, 9.161]`	Passed
zero control mean	`None`	`None`	Passed
summary sample 1	`[0.0, -28.1537, 39.186]`	`[0.0, -28.1537, 39.186]`	Passed
summary sample 2	`[-171.4798, -86.9258, -75.2184]`	`[-82.0, -86.9258, -75.2184]`	Failed
summary sample 3	`[-10.5361, -56.7354, 87.2201]`	`[-10.0, -56.7354, 87.2201]`	Failed
summary sample 4	`[-151.4128, -84.5965, -68.5785]`	`[-78.0, -84.5965, -68.5785]`	Failed

SHA-256 / a52fa37f7148fabeb7ed2ffa605d6ac97f0cd89b685952d89f6c1c858a4453f6

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(mean_c, se_c, mean_t, se_t, z):
    if mean_c <= 0 or mean_t <= 0:
        return None
    L = math.log(mean_t / mean_c)
    se = math.sqrt((se_t / mean_t) ** 2 + (se_c / mean_c) ** 2)
    return [round(math.exp(L) - 1, 4), round(100 * (math.exp(L - z * se) - 1), 4), round(100 * (math.exp(L + z * se) - 1), 4)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 1', [10.0, 0.2, 10.0, 2.0, 1.645], [0.0, -28.1537, 39.186]),
  ('summary sample 2', [10.0, 1.0, 1.8, 0.3, 1.645], [-82.0, -86.9258, -75.2184]),
  ('summary sample 3', [2.0, 0.5, 1.8, 0.5, 1.96], [-10.0, -56.7354, 87.2201]),
  ('summary sample 4', [50.0, 0.2, 11.0, 2.0, 1.96], [-78.0, -84.5965, -68.5785])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 6', [2.0, 0.2, 55.0, 2.0, 1.96], [2650.0, 2132.3268, 3287.7209]),
  ('summary sample 7', [50.0, 1.0, 10.0, 0.3, 1.645], [-80.0, -81.1517, -78.7779]),
  ('summary sample 8', [2.0, 0.2, 55.0, 2.0, 1.96], [2650.0, 2132.3268, 3287.7209]),
  ('summary sample 9', [50.0, 1.0, 11.0, 0.5, 1.645], [-78.0, -79.7257, -76.1274])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 11', [2.0, 0.5, 10.0, 0.5, 1.645], [400.0, 228.7224, 660.52]),
  ('summary sample 12', [2.0, 1.0, 10.0, 0.5, 1.645], [400.0, 118.7667, 1042.7699]),
  ('summary sample 13', [10.0, 1.0, 55.0, 0.3, 1.645], [450.0, 366.4607, 548.5005]),
  ('summary sample 14', [50.0, 0.2, 11.0, 0.3, 1.645], [-78.0, -78.9753, -76.9795])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 16', [50.0, 0.2, 10.0, 0.3, 1.645], [-80.0, -80.9714, -78.9791]),
  ('summary sample 17', [50.0, 0.2, 10.0, 0.5, 1.96], [-80.0, -81.8727, -77.9338]),
  ('summary sample 18', [2.0, 0.5, 11.0, 0.5, 1.96], [450.0, 234.2486, 805.0151]),
  ('summary sample 21', [50.0, 0.2, 1.8, 0.3, 1.645], [-96.4, -97.2635, -95.2641])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 21', [50.0, 0.2, 1.8, 0.3, 1.645], [-96.4, -97.2635, -95.2641]),
  ('summary sample 22', [10.0, 0.5, 55.0, 2.0, 1.96], [450.0, 387.2326, 520.8534]),
  ('summary sample 23', [2.0, 1.0, 10.0, 0.5, 1.96], [400.0, 86.7406, 1238.7557]),
  ('summary sample 28', [10.0, 0.5, 1.8, 0.3, 1.96], [-82.0, -87.2016, -74.6844])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
ten percent increase	`[0.1, 2.9448, 17.5387]`	`[10.0, 2.9448, 17.5387]`	Failed
decrease	`[-0.1, -51.3304, 66.4285]`	`[-10.0, -51.3304, 66.4285]`	Failed
no change	`[0.0, -8.3922, 9.161]`	`[0.0, -8.3922, 9.161]`	Passed
zero control mean	`None`	`None`	Passed
summary sample 1	`[0.0, -28.1537, 39.186]`	`[0.0, -28.1537, 39.186]`	Passed
summary sample 2	`[-0.82, -86.9258, -75.2184]`	`[-82.0, -86.9258, -75.2184]`	Failed
summary sample 3	`[-0.1, -56.7354, 87.2201]`	`[-10.0, -56.7354, 87.2201]`	Failed
summary sample 4	`[-0.78, -84.5965, -68.5785]`	`[-78.0, -84.5965, -68.5785]`	Failed

SHA-256 / 0a2705bfad5c888f9504ec36bddf4713b8f0e8be1873745f55bef878007e49f8

3 / The verified repair

Exit 0

"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(mean_c, se_c, mean_t, se_t, z):
    if mean_c <= 0 or mean_t <= 0:
        return None
    L = math.log(mean_t / mean_c)
    se = math.sqrt((se_t / mean_t) ** 2 + (se_c / mean_c) ** 2)
    return [round(100 * (math.exp(L) - 1), 4), round(100 * (math.exp(L - z * se) - 1), 4), round(100 * (math.exp(L + z * se) - 1), 4)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 1', [10.0, 0.2, 10.0, 2.0, 1.645], [0.0, -28.1537, 39.186]),
  ('summary sample 2', [10.0, 1.0, 1.8, 0.3, 1.645], [-82.0, -86.9258, -75.2184]),
  ('summary sample 3', [2.0, 0.5, 1.8, 0.5, 1.96], [-10.0, -56.7354, 87.2201]),
  ('summary sample 4', [50.0, 0.2, 11.0, 2.0, 1.96], [-78.0, -84.5965, -68.5785])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 6', [2.0, 0.2, 55.0, 2.0, 1.96], [2650.0, 2132.3268, 3287.7209]),
  ('summary sample 7', [50.0, 1.0, 10.0, 0.3, 1.645], [-80.0, -81.1517, -78.7779]),
  ('summary sample 8', [2.0, 0.2, 55.0, 2.0, 1.96], [2650.0, 2132.3268, 3287.7209]),
  ('summary sample 9', [50.0, 1.0, 11.0, 0.5, 1.645], [-78.0, -79.7257, -76.1274])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 11', [2.0, 0.5, 10.0, 0.5, 1.645], [400.0, 228.7224, 660.52]),
  ('summary sample 12', [2.0, 1.0, 10.0, 0.5, 1.645], [400.0, 118.7667, 1042.7699]),
  ('summary sample 13', [10.0, 1.0, 55.0, 0.3, 1.645], [450.0, 366.4607, 548.5005]),
  ('summary sample 14', [50.0, 0.2, 11.0, 0.3, 1.645], [-78.0, -78.9753, -76.9795])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 16', [50.0, 0.2, 10.0, 0.3, 1.645], [-80.0, -80.9714, -78.9791]),
  ('summary sample 17', [50.0, 0.2, 10.0, 0.5, 1.96], [-80.0, -81.8727, -77.9338]),
  ('summary sample 18', [2.0, 0.5, 11.0, 0.5, 1.96], [450.0, 234.2486, 805.0151]),
  ('summary sample 21', [50.0, 0.2, 1.8, 0.3, 1.645], [-96.4, -97.2635, -95.2641])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 21', [50.0, 0.2, 1.8, 0.3, 1.645], [-96.4, -97.2635, -95.2641]),
  ('summary sample 22', [10.0, 0.5, 55.0, 2.0, 1.96], [450.0, 387.2326, 520.8534]),
  ('summary sample 23', [2.0, 1.0, 10.0, 0.5, 1.96], [400.0, 86.7406, 1238.7557]),
  ('summary sample 28', [10.0, 0.5, 1.8, 0.3, 1.96], [-82.0, -87.2016, -74.6844])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
ten percent increase	`[10.0, 2.9448, 17.5387]`	`[10.0, 2.9448, 17.5387]`	Passed
decrease	`[-10.0, -51.3304, 66.4285]`	`[-10.0, -51.3304, 66.4285]`	Passed
no change	`[0.0, -8.3922, 9.161]`	`[0.0, -8.3922, 9.161]`	Passed
zero control mean	`None`	`None`	Passed
summary sample 1	`[0.0, -28.1537, 39.186]`	`[0.0, -28.1537, 39.186]`	Passed
summary sample 2	`[-82.0, -86.9258, -75.2184]`	`[-82.0, -86.9258, -75.2184]`	Passed
summary sample 3	`[-10.0, -56.7354, 87.2201]`	`[-10.0, -56.7354, 87.2201]`	Passed
summary sample 4	`[-78.0, -84.5965, -68.5785]`	`[-78.0, -84.5965, -68.5785]`	Passed

SHA-256 / ef44cb92b19f649d01fe36348c5bfb918be987836e609d92944a0692a4c152c7

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:49:00.375623+00:00.

Case digest / 3b4b12ea171c31263f8e8df758f7ee6452aa25095d9cde5154ce25a776025a84