FA-74551 / Experiment statistics / Open access

Post-exposure metric window: Exposed users without events disappear · case 01

Metric means are inflated because inactive exposed users are excluded.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

Totals are created lazily when an event qualifies.

VERIFIED REPAIR

Initialise every exposed user with a zero total.

Unsuccessful approach: Seeding only users with any event, even out of window, still drops fully silent users.

Case contract

events are [user, day, value]; exposures maps user -> first exposure day. A user's metric sums values of their events with exposure_day <= day < exposure_day + window_days. Every exposed user appears, even with no qualifying events; unexposed users are dropped. Return sorted [user, total].

Why this case matters

Counting pre-exposure activity dilutes effects; dropping silent users biases means upward.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(events, exposures, window_days):
    totals = {}
    for user, day, value in events:
        if user not in exposures:
            continue
        start = exposures[user]
        if start <= day < start + window_days:
            totals[user] = totals.get(user, 0) + value
    return sorted([u, t] for u, t in totals.items())
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),
  ('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),
  ('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),
  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),
  ('event log sample 1',
   [[['a', 7, 1], ['b', 7, 1], ['d', 10, 1], ['e', 10, 1], ['b', 5, 1], ['c', 11, 1]],
    {'c': 5, 'a': 0, 'b': 4},
    1],
   [['a', 0], ['b', 0], ['c', 0]]),
  ('event log sample 2',
   [[['c', 1, 1], ['a', 6, 2], ['a', 2, 5]], {'c': 3, 'd': 3}, 7],
   [['c', 0], ['d', 0]])],
 [('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),
  ('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),
  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),
  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),
  ('event log sample 7',
   [[['d', 4, 1], ['a', 5, 2], ['a', 10, 5], ['b', 3, 2], ['a', 3, 2], ['c', 0, 1]], {'b': 5, 'a': 5}, 3],
   [['a', 2], ['b', 0]]),
  ('event log sample 10', [[], {'b': 2, 'a': 4, 'c': 5}, 1], [['a', 0], ['b', 0], ['c', 0]])],
 [('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),
  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),
  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),
  ('event log sample 11', [[['c', 7, 2], ['b', 3, 1]], {'a': 4, 'd': 2}, 1], [['a', 0], ['d', 0]]),
  ('event log sample 14',
   [[['d', 2, 2], ['d', 2, 1], ['b', 3, 2], ['a', 8, 2], ['d', 9, 1], ['d', 6, 2]], {'d': 0, 'c': 3}, 3],
   [['c', 0], ['d', 3]]),
  ('event log sample 19', [[['b', 3, 2], ['e', 1, 2], ['d', 8, 1], ['b', 4, 2]], {'c': 4}, 3], [['c', 0]])],
 [('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),
  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),
  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),
  ('event log sample 16',
   [[['d', 0, 5], ['b', 3, 2], ['d', 1, 1], ['d', 2, 1]], {'c': 1, 'b': 5}, 1],
   [['b', 0], ['c', 0]]),
  ('event log sample 23',
   [[['b', 1, 2], ['d', 11, 2], ['c', 10, 1], ['a', 10, 5], ['b', 5, 1]], {'c': 2}, 3],
   [['c', 0]]),
  ('event log sample 28',
   [[['a', 10, 1], ['d', 9, 1], ['d', 3, 1], ['d', 6, 5]], {'b': 1, 'd': 0, 'c': 1}, 7],
   [['b', 0], ['c', 0], ['d', 6]])],
 [('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),
  ('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),
  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),
  ('event log sample 21', [[], {'d': 5, 'c': 2}, 7], [['c', 0], ['d', 0]]),
  ('event log sample 30',
   [[['e', 7, 5],
     ['a', 6, 5],
     ['e', 3, 5],
     ['a', 3, 5],
     ['a', 3, 1],
     ['c', 11, 2],
     ['d', 8, 2],
     ['b', 10, 1]],
    {'d': 5},
    3],
   [['d', 0]]),
  ('event log sample 42',
   [[['d', 8, 2], ['e', 10, 2]], {'d': 4, 'a': 0, 'b': 0}, 1],
   [['a', 0], ['b', 0], ['d', 0]])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
pre-exposure activity is excluded	`[['a', 2]]`	`[['a', 2]]`	Passed
exposure day itself counts	`[['a', 4]]`	`[['a', 4]]`	Passed
window end is exclusive	`[['a', 1]]`	`[['a', 1]]`	Passed
last day inside the window counts	`[['b', 2]]`	`[['b', 2]]`	Passed
exposed user with no events reports zero	`[['a', 1]]`	`[['a', 1], ['b', 0]]`	Failed
unexposed users are dropped	`[]`	`[['a', 0]]`	Failed
event log sample 1	`[]`	`[['a', 0], ['b', 0], ['c', 0]]`	Failed
event log sample 2	`[]`	`[['c', 0], ['d', 0]]`	Failed

SHA-256 / f638e44f575c65b9fe4497391561aadd540ac0ff64b6dcacf391b52b1173e6ae

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(events, exposures, window_days):
    totals = {u: 0 for u in exposures if any(e[0] == u for e in events)}
    for user, day, value in events:
        if user not in exposures:
            continue
        start = exposures[user]
        if start <= day < start + window_days:
            totals[user] = totals.get(user, 0) + value
    return sorted([u, t] for u, t in totals.items())
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),
  ('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),
  ('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),
  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),
  ('event log sample 1',
   [[['a', 7, 1], ['b', 7, 1], ['d', 10, 1], ['e', 10, 1], ['b', 5, 1], ['c', 11, 1]],
    {'c': 5, 'a': 0, 'b': 4},
    1],
   [['a', 0], ['b', 0], ['c', 0]]),
  ('event log sample 2',
   [[['c', 1, 1], ['a', 6, 2], ['a', 2, 5]], {'c': 3, 'd': 3}, 7],
   [['c', 0], ['d', 0]])],
 [('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),
  ('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),
  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),
  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),
  ('event log sample 7',
   [[['d', 4, 1], ['a', 5, 2], ['a', 10, 5], ['b', 3, 2], ['a', 3, 2], ['c', 0, 1]], {'b': 5, 'a': 5}, 3],
   [['a', 2], ['b', 0]]),
  ('event log sample 10', [[], {'b': 2, 'a': 4, 'c': 5}, 1], [['a', 0], ['b', 0], ['c', 0]])],
 [('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),
  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),
  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),
  ('event log sample 11', [[['c', 7, 2], ['b', 3, 1]], {'a': 4, 'd': 2}, 1], [['a', 0], ['d', 0]]),
  ('event log sample 14',
   [[['d', 2, 2], ['d', 2, 1], ['b', 3, 2], ['a', 8, 2], ['d', 9, 1], ['d', 6, 2]], {'d': 0, 'c': 3}, 3],
   [['c', 0], ['d', 3]]),
  ('event log sample 19', [[['b', 3, 2], ['e', 1, 2], ['d', 8, 1], ['b', 4, 2]], {'c': 4}, 3], [['c', 0]])],
 [('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),
  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),
  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),
  ('event log sample 16',
   [[['d', 0, 5], ['b', 3, 2], ['d', 1, 1], ['d', 2, 1]], {'c': 1, 'b': 5}, 1],
   [['b', 0], ['c', 0]]),
  ('event log sample 23',
   [[['b', 1, 2], ['d', 11, 2], ['c', 10, 1], ['a', 10, 5], ['b', 5, 1]], {'c': 2}, 3],
   [['c', 0]]),
  ('event log sample 28',
   [[['a', 10, 1], ['d', 9, 1], ['d', 3, 1], ['d', 6, 5]], {'b': 1, 'd': 0, 'c': 1}, 7],
   [['b', 0], ['c', 0], ['d', 6]])],
 [('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),
  ('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),
  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),
  ('event log sample 21', [[], {'d': 5, 'c': 2}, 7], [['c', 0], ['d', 0]]),
  ('event log sample 30',
   [[['e', 7, 5],
     ['a', 6, 5],
     ['e', 3, 5],
     ['a', 3, 5],
     ['a', 3, 1],
     ['c', 11, 2],
     ['d', 8, 2],
     ['b', 10, 1]],
    {'d': 5},
    3],
   [['d', 0]]),
  ('event log sample 42',
   [[['d', 8, 2], ['e', 10, 2]], {'d': 4, 'a': 0, 'b': 0}, 1],
   [['a', 0], ['b', 0], ['d', 0]])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
pre-exposure activity is excluded	`[['a', 2]]`	`[['a', 2]]`	Passed
exposure day itself counts	`[['a', 4]]`	`[['a', 4]]`	Passed
window end is exclusive	`[['a', 1]]`	`[['a', 1]]`	Passed
last day inside the window counts	`[['b', 2]]`	`[['b', 2]]`	Passed
exposed user with no events reports zero	`[['a', 1]]`	`[['a', 1], ['b', 0]]`	Failed
unexposed users are dropped	`[]`	`[['a', 0]]`	Failed
event log sample 1	`[['a', 0], ['b', 0], ['c', 0]]`	`[['a', 0], ['b', 0], ['c', 0]]`	Passed
event log sample 2	`[['c', 0]]`	`[['c', 0], ['d', 0]]`	Failed

SHA-256 / 13879a0d363cd7eacea9a33bc41283186b3026d8ecf0cdb9b311ff9554bad1c4

3 / The verified repair

Exit 0

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(events, exposures, window_days):
    totals = {u: 0 for u in exposures}
    for user, day, value in events:
        if user not in exposures:
            continue
        start = exposures[user]
        if start <= day < start + window_days:
            totals[user] += value
    return sorted([u, t] for u, t in totals.items())
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),
  ('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),
  ('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),
  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),
  ('event log sample 1',
   [[['a', 7, 1], ['b', 7, 1], ['d', 10, 1], ['e', 10, 1], ['b', 5, 1], ['c', 11, 1]],
    {'c': 5, 'a': 0, 'b': 4},
    1],
   [['a', 0], ['b', 0], ['c', 0]]),
  ('event log sample 2',
   [[['c', 1, 1], ['a', 6, 2], ['a', 2, 5]], {'c': 3, 'd': 3}, 7],
   [['c', 0], ['d', 0]])],
 [('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),
  ('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),
  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),
  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),
  ('event log sample 7',
   [[['d', 4, 1], ['a', 5, 2], ['a', 10, 5], ['b', 3, 2], ['a', 3, 2], ['c', 0, 1]], {'b': 5, 'a': 5}, 3],
   [['a', 2], ['b', 0]]),
  ('event log sample 10', [[], {'b': 2, 'a': 4, 'c': 5}, 1], [['a', 0], ['b', 0], ['c', 0]])],
 [('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),
  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),
  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),
  ('event log sample 11', [[['c', 7, 2], ['b', 3, 1]], {'a': 4, 'd': 2}, 1], [['a', 0], ['d', 0]]),
  ('event log sample 14',
   [[['d', 2, 2], ['d', 2, 1], ['b', 3, 2], ['a', 8, 2], ['d', 9, 1], ['d', 6, 2]], {'d': 0, 'c': 3}, 3],
   [['c', 0], ['d', 3]]),
  ('event log sample 19', [[['b', 3, 2], ['e', 1, 2], ['d', 8, 1], ['b', 4, 2]], {'c': 4}, 3], [['c', 0]])],
 [('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),
  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),
  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),
  ('event log sample 16',
   [[['d', 0, 5], ['b', 3, 2], ['d', 1, 1], ['d', 2, 1]], {'c': 1, 'b': 5}, 1],
   [['b', 0], ['c', 0]]),
  ('event log sample 23',
   [[['b', 1, 2], ['d', 11, 2], ['c', 10, 1], ['a', 10, 5], ['b', 5, 1]], {'c': 2}, 3],
   [['c', 0]]),
  ('event log sample 28',
   [[['a', 10, 1], ['d', 9, 1], ['d', 3, 1], ['d', 6, 5]], {'b': 1, 'd': 0, 'c': 1}, 7],
   [['b', 0], ['c', 0], ['d', 6]])],
 [('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),
  ('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),
  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),
  ('event log sample 21', [[], {'d': 5, 'c': 2}, 7], [['c', 0], ['d', 0]]),
  ('event log sample 30',
   [[['e', 7, 5],
     ['a', 6, 5],
     ['e', 3, 5],
     ['a', 3, 5],
     ['a', 3, 1],
     ['c', 11, 2],
     ['d', 8, 2],
     ['b', 10, 1]],
    {'d': 5},
    3],
   [['d', 0]]),
  ('event log sample 42',
   [[['d', 8, 2], ['e', 10, 2]], {'d': 4, 'a': 0, 'b': 0}, 1],
   [['a', 0], ['b', 0], ['d', 0]])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
pre-exposure activity is excluded	`[['a', 2]]`	`[['a', 2]]`	Passed
exposure day itself counts	`[['a', 4]]`	`[['a', 4]]`	Passed
window end is exclusive	`[['a', 1]]`	`[['a', 1]]`	Passed
last day inside the window counts	`[['b', 2]]`	`[['b', 2]]`	Passed
exposed user with no events reports zero	`[['a', 1], ['b', 0]]`	`[['a', 1], ['b', 0]]`	Passed
unexposed users are dropped	`[['a', 0]]`	`[['a', 0]]`	Passed
event log sample 1	`[['a', 0], ['b', 0], ['c', 0]]`	`[['a', 0], ['b', 0], ['c', 0]]`	Passed
event log sample 2	`[['c', 0], ['d', 0]]`	`[['c', 0], ['d', 0]]`	Passed

SHA-256 / b596d9afc734a189197bd273bb55d79805d3df00e6d11f3d3a188be337aa2afa

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:57.863238+00:00.

Case digest / e7ab131307aa1d5ec854f4d91c100ad17011eec0adaf4508b0de1116e1536415