FA-74826 / Experiment statistics / Open access

Percent change log-ratio interval: Interval ends stay on the log scale · case 01

Bounds are reported as log points while the estimate is a percent change.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The interval ends are 100 (L -/+ z se) without exponentiating.

VERIFIED REPAIR

Exponentiate each bound before converting to percent change.

Unsuccessful approach: Exponentiating only the centre and adding z se linearly loses the asymmetry.

Case contract

L = ln(mean_t / mean_c) with standard error sqrt((se_t / mean_t)^2 + (se_c / mean_c)^2). Report the percent change 100 (e^L - 1) and the interval 100 (e^(L -/+ z se) - 1), each rounded to 4. Nonpositive means -> None.

Why this case matters

Log-ratio intervals keep percent-change bounds asymmetric and positive-definite.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(mean_c, se_c, mean_t, se_t, z):
    if mean_c <= 0 or mean_t <= 0:
        return None
    L = math.log(mean_t / mean_c)
    se = math.sqrt((se_t / mean_t) ** 2 + (se_c / mean_c) ** 2)
    return [round(100 * (math.exp(L) - 1), 4), round(100 * (L - z * se), 4), round(100 * (L + z * se), 4)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 1', [10.0, 0.2, 10.0, 2.0, 1.645], [0.0, -28.1537, 39.186]),
  ('summary sample 2', [10.0, 1.0, 1.8, 0.3, 1.645], [-82.0, -86.9258, -75.2184]),
  ('summary sample 3', [2.0, 0.5, 1.8, 0.5, 1.96], [-10.0, -56.7354, 87.2201]),
  ('summary sample 4', [50.0, 0.2, 11.0, 2.0, 1.96], [-78.0, -84.5965, -68.5785])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 5', [2.0, 1.0, 10.0, 0.3, 1.645], [400.0, 119.3413, 1039.7763]),
  ('summary sample 6', [2.0, 0.2, 55.0, 2.0, 1.96], [2650.0, 2132.3268, 3287.7209]),
  ('summary sample 7', [50.0, 1.0, 10.0, 0.3, 1.645], [-80.0, -81.1517, -78.7779]),
  ('summary sample 8', [2.0, 0.2, 55.0, 2.0, 1.96], [2650.0, 2132.3268, 3287.7209])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 11', [2.0, 0.5, 10.0, 0.5, 1.645], [400.0, 228.7224, 660.52]),
  ('summary sample 12', [2.0, 1.0, 10.0, 0.5, 1.645], [400.0, 118.7667, 1042.7699]),
  ('summary sample 13', [10.0, 1.0, 55.0, 0.3, 1.645], [450.0, 366.4607, 548.5005]),
  ('summary sample 14', [50.0, 0.2, 11.0, 0.3, 1.645], [-78.0, -78.9753, -76.9795])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 16', [50.0, 0.2, 10.0, 0.3, 1.645], [-80.0, -80.9714, -78.9791]),
  ('summary sample 17', [50.0, 0.2, 10.0, 0.5, 1.96], [-80.0, -81.8727, -77.9338]),
  ('summary sample 18', [2.0, 0.5, 11.0, 0.5, 1.96], [450.0, 234.2486, 805.0151]),
  ('summary sample 19', [2.0, 1.0, 10.0, 0.3, 1.96], [400.0, 87.3251, 1234.5781])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 21', [50.0, 0.2, 1.8, 0.3, 1.645], [-96.4, -97.2635, -95.2641]),
  ('summary sample 22', [10.0, 0.5, 55.0, 2.0, 1.96], [450.0, 387.2326, 520.8534]),
  ('summary sample 23', [2.0, 1.0, 10.0, 0.5, 1.96], [400.0, 86.7406, 1238.7557]),
  ('summary sample 26', [50.0, 0.2, 55.0, 0.5, 1.96], [10.0, 7.8794, 12.1623])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
ten percent increase	`[10.0, 2.9023, 16.1598]`	`[10.0, 2.9448, 17.5387]`	Failed
decrease	`[-10.0, -72.0116, 50.9395]`	`[-10.0, -51.3304, 66.4285]`	Failed
no change	`[0.0, -8.7654, 8.7654]`	`[0.0, -8.3922, 9.161]`	Failed
zero control mean	`None`	`None`	Passed
summary sample 1	`[0.0, -33.0641, 33.0641]`	`[0.0, -28.1537, 39.186]`	Failed
summary sample 2	`[-82.0, -203.4529, -139.5068]`	`[-82.0, -86.9258, -75.2184]`	Failed
summary sample 3	`[-10.0, -83.7836, 62.7115]`	`[-10.0, -56.7354, 87.2201]`	Failed
summary sample 4	`[-78.0, -187.0578, -115.7678]`	`[-78.0, -84.5965, -68.5785]`	Failed

SHA-256 / 564eac7de6a8706c61d6c6546f08787c6c822bdd5739a9b6bcd4d5e83b70e608

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(mean_c, se_c, mean_t, se_t, z):
    if mean_c <= 0 or mean_t <= 0:
        return None
    L = math.log(mean_t / mean_c)
    se = math.sqrt((se_t / mean_t) ** 2 + (se_c / mean_c) ** 2)
    return [round(100 * (math.exp(L) - 1), 4), round(100 * (math.exp(L) - 1 - z * se), 4), round(100 * (math.exp(L) - 1 + z * se), 4)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 1', [10.0, 0.2, 10.0, 2.0, 1.645], [0.0, -28.1537, 39.186]),
  ('summary sample 2', [10.0, 1.0, 1.8, 0.3, 1.645], [-82.0, -86.9258, -75.2184]),
  ('summary sample 3', [2.0, 0.5, 1.8, 0.5, 1.96], [-10.0, -56.7354, 87.2201]),
  ('summary sample 4', [50.0, 0.2, 11.0, 2.0, 1.96], [-78.0, -84.5965, -68.5785])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 5', [2.0, 1.0, 10.0, 0.3, 1.645], [400.0, 119.3413, 1039.7763]),
  ('summary sample 6', [2.0, 0.2, 55.0, 2.0, 1.96], [2650.0, 2132.3268, 3287.7209]),
  ('summary sample 7', [50.0, 1.0, 10.0, 0.3, 1.645], [-80.0, -81.1517, -78.7779]),
  ('summary sample 8', [2.0, 0.2, 55.0, 2.0, 1.96], [2650.0, 2132.3268, 3287.7209])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 11', [2.0, 0.5, 10.0, 0.5, 1.645], [400.0, 228.7224, 660.52]),
  ('summary sample 12', [2.0, 1.0, 10.0, 0.5, 1.645], [400.0, 118.7667, 1042.7699]),
  ('summary sample 13', [10.0, 1.0, 55.0, 0.3, 1.645], [450.0, 366.4607, 548.5005]),
  ('summary sample 14', [50.0, 0.2, 11.0, 0.3, 1.645], [-78.0, -78.9753, -76.9795])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 16', [50.0, 0.2, 10.0, 0.3, 1.645], [-80.0, -80.9714, -78.9791]),
  ('summary sample 17', [50.0, 0.2, 10.0, 0.5, 1.96], [-80.0, -81.8727, -77.9338]),
  ('summary sample 18', [2.0, 0.5, 11.0, 0.5, 1.96], [450.0, 234.2486, 805.0151]),
  ('summary sample 19', [2.0, 1.0, 10.0, 0.3, 1.96], [400.0, 87.3251, 1234.5781])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 21', [50.0, 0.2, 1.8, 0.3, 1.645], [-96.4, -97.2635, -95.2641]),
  ('summary sample 22', [10.0, 0.5, 55.0, 2.0, 1.96], [450.0, 387.2326, 520.8534]),
  ('summary sample 23', [2.0, 1.0, 10.0, 0.5, 1.96], [400.0, 86.7406, 1238.7557]),
  ('summary sample 26', [50.0, 0.2, 55.0, 0.5, 1.96], [10.0, 7.8794, 12.1623])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
ten percent increase	`[10.0, 3.3713, 16.6287]`	`[10.0, 2.9448, 17.5387]`	Failed
decrease	`[-10.0, -71.4756, 51.4756]`	`[-10.0, -51.3304, 66.4285]`	Failed
no change	`[0.0, -8.7654, 8.7654]`	`[0.0, -8.3922, 9.161]`	Failed
zero control mean	`None`	`None`	Passed
summary sample 1	`[0.0, -33.0641, 33.0641]`	`[0.0, -28.1537, 39.186]`	Failed
summary sample 2	`[-82.0, -113.9731, -50.0269]`	`[-82.0, -86.9258, -75.2184]`	Failed
summary sample 3	`[-10.0, -83.2475, 63.2475]`	`[-10.0, -56.7354, 87.2201]`	Failed
summary sample 4	`[-78.0, -113.645, -42.355]`	`[-78.0, -84.5965, -68.5785]`	Failed

SHA-256 / 524a4643d637e6f7367e7c3e7fb46093d3d716c1cd8b01b01e137eb99e6271f0

3 / The verified repair

Exit 0

"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(mean_c, se_c, mean_t, se_t, z):
    if mean_c <= 0 or mean_t <= 0:
        return None
    L = math.log(mean_t / mean_c)
    se = math.sqrt((se_t / mean_t) ** 2 + (se_c / mean_c) ** 2)
    return [round(100 * (math.exp(L) - 1), 4), round(100 * (math.exp(L - z * se) - 1), 4), round(100 * (math.exp(L + z * se) - 1), 4)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 1', [10.0, 0.2, 10.0, 2.0, 1.645], [0.0, -28.1537, 39.186]),
  ('summary sample 2', [10.0, 1.0, 1.8, 0.3, 1.645], [-82.0, -86.9258, -75.2184]),
  ('summary sample 3', [2.0, 0.5, 1.8, 0.5, 1.96], [-10.0, -56.7354, 87.2201]),
  ('summary sample 4', [50.0, 0.2, 11.0, 2.0, 1.96], [-78.0, -84.5965, -68.5785])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 5', [2.0, 1.0, 10.0, 0.3, 1.645], [400.0, 119.3413, 1039.7763]),
  ('summary sample 6', [2.0, 0.2, 55.0, 2.0, 1.96], [2650.0, 2132.3268, 3287.7209]),
  ('summary sample 7', [50.0, 1.0, 10.0, 0.3, 1.645], [-80.0, -81.1517, -78.7779]),
  ('summary sample 8', [2.0, 0.2, 55.0, 2.0, 1.96], [2650.0, 2132.3268, 3287.7209])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 11', [2.0, 0.5, 10.0, 0.5, 1.645], [400.0, 228.7224, 660.52]),
  ('summary sample 12', [2.0, 1.0, 10.0, 0.5, 1.645], [400.0, 118.7667, 1042.7699]),
  ('summary sample 13', [10.0, 1.0, 55.0, 0.3, 1.645], [450.0, 366.4607, 548.5005]),
  ('summary sample 14', [50.0, 0.2, 11.0, 0.3, 1.645], [-78.0, -78.9753, -76.9795])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 16', [50.0, 0.2, 10.0, 0.3, 1.645], [-80.0, -80.9714, -78.9791]),
  ('summary sample 17', [50.0, 0.2, 10.0, 0.5, 1.96], [-80.0, -81.8727, -77.9338]),
  ('summary sample 18', [2.0, 0.5, 11.0, 0.5, 1.96], [450.0, 234.2486, 805.0151]),
  ('summary sample 19', [2.0, 1.0, 10.0, 0.3, 1.96], [400.0, 87.3251, 1234.5781])],
 [('ten percent increase', [10.0, 0.2, 11.0, 0.3, 1.96], [10.0, 2.9448, 17.5387]),
  ('decrease', [2.0, 0.5, 1.8, 0.5, 1.645], [-10.0, -51.3304, 66.4285]),
  ('no change', [50.0, 1.0, 50.0, 2.0, 1.96], [0.0, -8.3922, 9.161]),
  ('zero control mean', [0.0, 1.0, 2.0, 1.0, 1.96], None),
  ('summary sample 21', [50.0, 0.2, 1.8, 0.3, 1.645], [-96.4, -97.2635, -95.2641]),
  ('summary sample 22', [10.0, 0.5, 55.0, 2.0, 1.96], [450.0, 387.2326, 520.8534]),
  ('summary sample 23', [2.0, 1.0, 10.0, 0.5, 1.96], [400.0, 86.7406, 1238.7557]),
  ('summary sample 26', [50.0, 0.2, 55.0, 0.5, 1.96], [10.0, 7.8794, 12.1623])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
ten percent increase	`[10.0, 2.9448, 17.5387]`	`[10.0, 2.9448, 17.5387]`	Passed
decrease	`[-10.0, -51.3304, 66.4285]`	`[-10.0, -51.3304, 66.4285]`	Passed
no change	`[0.0, -8.3922, 9.161]`	`[0.0, -8.3922, 9.161]`	Passed
zero control mean	`None`	`None`	Passed
summary sample 1	`[0.0, -28.1537, 39.186]`	`[0.0, -28.1537, 39.186]`	Passed
summary sample 2	`[-82.0, -86.9258, -75.2184]`	`[-82.0, -86.9258, -75.2184]`	Passed
summary sample 3	`[-10.0, -56.7354, 87.2201]`	`[-10.0, -56.7354, 87.2201]`	Passed
summary sample 4	`[-78.0, -84.5965, -68.5785]`	`[-78.0, -84.5965, -68.5785]`	Passed

SHA-256 / 6337fa250e1bacf768f2e268648201f415d8c3a8ee17ee658bbfc024a06c4f70

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:49:00.314186+00:00.

Case digest / 8c07dd170706df40e73b6ddd1966b6f61c7fdd9235e5b46517e4d71b1ac29a43