FAILURE MAP
← Case archive

FA-74556 / Experiment statistics / Open access

Post-exposure metric window: All users share the earliest exposure as window start · case 01

Late-exposed users contribute almost nothing because their window closed before they arrived.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The window start is min(exposures.values()) instead of the user's own exposure day.

THE FAILURE

The window start is min(exposures.values()) instead of the user's own exposure day.

Unsuccessful approach: Anchoring at the latest exposure counts pre-exposure activity for early users.

Case contract

events are [user, day, value]; exposures maps user -> first exposure day. A user's metric sums values of their events with exposure_day <= day < exposure_day + window_days. Every exposed user appears, even with no qualifying events; unexposed users are dropped. Return sorted [user, total].

Why this case matters

Counting pre-exposure activity dilutes effects; dropping silent users biases means upward.

1 / The failure

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(events, exposures, window_days):
    totals = {u: 0 for u in exposures}
    for user, day, value in events:
        if user not in exposures:
            continue
        start = min(exposures.values())
        if start <= day < start + window_days:
            totals[user] += value
    return sorted([u, t] for u, t in totals.items())
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),
  ('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),
  ('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),
  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),
  ('event log sample 1',
   [[['a', 7, 1], ['b', 7, 1], ['d', 10, 1], ['e', 10, 1], ['b', 5, 1], ['c', 11, 1]],
    {'c': 5, 'a': 0, 'b': 4},
    1],
   [['a', 0], ['b', 0], ['c', 0]]),
  ('event log sample 2',
   [[['c', 1, 1], ['a', 6, 2], ['a', 2, 5]], {'c': 3, 'd': 3}, 7],
   [['c', 0], ['d', 0]])],
 [('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),
  ('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),
  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),
  ('event log sample 6',
   [[['d', 1, 1], ['e', 8, 5], ['c', 5, 2], ['e', 1, 2], ['c', 9, 2], ['a', 1, 2]], {'d': 0}, 3],
   [['d', 1]]),
  ('event log sample 7',
   [[['d', 4, 1], ['a', 5, 2], ['a', 10, 5], ['b', 3, 2], ['a', 3, 2], ['c', 0, 1]], {'b': 5, 'a': 5}, 3],
   [['a', 2], ['b', 0]]),
  ('event log sample 43',
   [[['e', 7, 2], ['e', 1, 2], ['a', 3, 2], ['e', 3, 1]], {'b': 4, 'a': 3, 'd': 3}, 1],
   [['a', 2], ['b', 0], ['d', 0]])],
 [('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),
  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),
  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),
  ('event log sample 11', [[['c', 7, 2], ['b', 3, 1]], {'a': 4, 'd': 2}, 1], [['a', 0], ['d', 0]]),
  ('event log sample 12',
   [[['c', 2, 5], ['c', 10, 5], ['a', 9, 1]], {'a': 3, 'd': 0, 'b': 3}, 1],
   [['a', 0], ['b', 0], ['d', 0]]),
  ('event log sample 14',
   [[['d', 2, 2], ['d', 2, 1], ['b', 3, 2], ['a', 8, 2], ['d', 9, 1], ['d', 6, 2]], {'d': 0, 'c': 3}, 3],
   [['c', 0], ['d', 3]])],
 [('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),
  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),
  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),
  ('event log sample 16',
   [[['d', 0, 5], ['b', 3, 2], ['d', 1, 1], ['d', 2, 1]], {'c': 1, 'b': 5}, 1],
   [['b', 0], ['c', 0]]),
  ('event log sample 17',
   [[['d', 4, 2], ['a', 5, 5], ['b', 0, 2], ['e', 7, 1], ['a', 11, 1], ['e', 4, 1], ['a', 8, 1]],
    {'c': 5, 'a': 4},
    7],
   [['a', 6], ['c', 0]]),
  ('event log sample 59',
   [[['a', 2, 2],
     ['c', 5, 1],
     ['d', 0, 2],
     ['d', 11, 5],
     ['e', 10, 1],
     ['e', 1, 5],
     ['c', 9, 5],
     ['b', 4, 5]],
    {'a': 1, 'b': 4, 'c': 3},
    7],
   [['a', 2], ['b', 5], ['c', 6]])],
 [('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),
  ('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),
  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),
  ('event log sample 21', [[], {'d': 5, 'c': 2}, 7], [['c', 0], ['d', 0]]),
  ('event log sample 22', [[['e', 3, 2]], {'c': 1, 'b': 4}, 1], [['b', 0], ['c', 0]]),
  ('event log sample 40',
   [[['a', 8, 2], ['c', 8, 1], ['b', 8, 1], ['d', 1, 2], ['a', 2, 5]], {'d': 1, 'b': 2}, 7],
   [['b', 1], ['d', 2]])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
pre-exposure activity is excluded[['a', 2]][['a', 2]]Passed
exposure day itself counts[['a', 4]][['a', 4]]Passed
window end is exclusive[['a', 1]][['a', 1]]Passed
last day inside the window counts[['b', 2]][['b', 2]]Passed
exposed user with no events reports zero[['a', 1], ['b', 0]][['a', 1], ['b', 0]]Passed
each user has their own window[['a', 1], ['b', 0]][['a', 1], ['b', 1]]Failed
event log sample 1[['a', 0], ['b', 0], ['c', 0]][['a', 0], ['b', 0], ['c', 0]]Passed
event log sample 2[['c', 0], ['d', 0]][['c', 0], ['d', 0]]Passed

SHA-256 / cd22fb1c1253b120e7f877bc3df7e310e5c0f82ea06fa5d611503431b03f7cfa

2 / The unsuccessful fix

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(events, exposures, window_days):
    totals = {u: 0 for u in exposures}
    for user, day, value in events:
        if user not in exposures:
            continue
        start = max(exposures.values())
        if start <= day < start + window_days:
            totals[user] += value
    return sorted([u, t] for u, t in totals.items())
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),
  ('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),
  ('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),
  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),
  ('event log sample 1',
   [[['a', 7, 1], ['b', 7, 1], ['d', 10, 1], ['e', 10, 1], ['b', 5, 1], ['c', 11, 1]],
    {'c': 5, 'a': 0, 'b': 4},
    1],
   [['a', 0], ['b', 0], ['c', 0]]),
  ('event log sample 2',
   [[['c', 1, 1], ['a', 6, 2], ['a', 2, 5]], {'c': 3, 'd': 3}, 7],
   [['c', 0], ['d', 0]])],
 [('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),
  ('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),
  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),
  ('event log sample 6',
   [[['d', 1, 1], ['e', 8, 5], ['c', 5, 2], ['e', 1, 2], ['c', 9, 2], ['a', 1, 2]], {'d': 0}, 3],
   [['d', 1]]),
  ('event log sample 7',
   [[['d', 4, 1], ['a', 5, 2], ['a', 10, 5], ['b', 3, 2], ['a', 3, 2], ['c', 0, 1]], {'b': 5, 'a': 5}, 3],
   [['a', 2], ['b', 0]]),
  ('event log sample 43',
   [[['e', 7, 2], ['e', 1, 2], ['a', 3, 2], ['e', 3, 1]], {'b': 4, 'a': 3, 'd': 3}, 1],
   [['a', 2], ['b', 0], ['d', 0]])],
 [('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),
  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),
  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),
  ('event log sample 11', [[['c', 7, 2], ['b', 3, 1]], {'a': 4, 'd': 2}, 1], [['a', 0], ['d', 0]]),
  ('event log sample 12',
   [[['c', 2, 5], ['c', 10, 5], ['a', 9, 1]], {'a': 3, 'd': 0, 'b': 3}, 1],
   [['a', 0], ['b', 0], ['d', 0]]),
  ('event log sample 14',
   [[['d', 2, 2], ['d', 2, 1], ['b', 3, 2], ['a', 8, 2], ['d', 9, 1], ['d', 6, 2]], {'d': 0, 'c': 3}, 3],
   [['c', 0], ['d', 3]])],
 [('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),
  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),
  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),
  ('event log sample 16',
   [[['d', 0, 5], ['b', 3, 2], ['d', 1, 1], ['d', 2, 1]], {'c': 1, 'b': 5}, 1],
   [['b', 0], ['c', 0]]),
  ('event log sample 17',
   [[['d', 4, 2], ['a', 5, 5], ['b', 0, 2], ['e', 7, 1], ['a', 11, 1], ['e', 4, 1], ['a', 8, 1]],
    {'c': 5, 'a': 4},
    7],
   [['a', 6], ['c', 0]]),
  ('event log sample 59',
   [[['a', 2, 2],
     ['c', 5, 1],
     ['d', 0, 2],
     ['d', 11, 5],
     ['e', 10, 1],
     ['e', 1, 5],
     ['c', 9, 5],
     ['b', 4, 5]],
    {'a': 1, 'b': 4, 'c': 3},
    7],
   [['a', 2], ['b', 5], ['c', 6]])],
 [('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),
  ('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),
  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),
  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),
  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),
  ('event log sample 21', [[], {'d': 5, 'c': 2}, 7], [['c', 0], ['d', 0]]),
  ('event log sample 22', [[['e', 3, 2]], {'c': 1, 'b': 4}, 1], [['b', 0], ['c', 0]]),
  ('event log sample 40',
   [[['a', 8, 2], ['c', 8, 1], ['b', 8, 1], ['d', 1, 2], ['a', 2, 5]], {'d': 1, 'b': 2}, 7],
   [['b', 1], ['d', 2]])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
pre-exposure activity is excluded[['a', 2]][['a', 2]]Passed
exposure day itself counts[['a', 4]][['a', 4]]Passed
window end is exclusive[['a', 1]][['a', 1]]Passed
last day inside the window counts[['b', 2]][['b', 2]]Passed
exposed user with no events reports zero[['a', 0], ['b', 0]][['a', 1], ['b', 0]]Failed
each user has their own window[['a', 0], ['b', 1]][['a', 1], ['b', 1]]Failed
event log sample 1[['a', 0], ['b', 1], ['c', 0]][['a', 0], ['b', 0], ['c', 0]]Failed
event log sample 2[['c', 0], ['d', 0]][['c', 0], ['d', 0]]Passed

SHA-256 / 0d2468ef23b8da5836d3d58100320dd1868fc4bad341b07473d322ffa8dab295

HELD IN THE MEMBER ARCHIVE

The verified repair and its recorded checks are member-only.

This mechanism has 8 recorded checks per implementation. The open-access tier publishes the failure and the unsuccessful fix; the repaired source that passes every check, and the observations that prove it, are available to members.

Every case sharing this mechanism uses the same contract and the same repair, so this one record is held back for all of them.

Member access is invitation-based. Sign in with your invited account to inspect the repair.

Sign in to the archive ↗

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:57.896215+00:00.

Case digest / 4eb15bd0e61db7d440d8be34dd94229315584cf689f8b5577aa10d4f7a93d5f4