{"abstract":"Treatment effects are diluted by behaviour that happened before users saw the change.","category":"Experiment statistics","checks":8,"contract":"events are [user, day, value]; exposures maps user -> first exposure day. A user's metric sums values of their events with exposure_day <= day < exposure_day + window_days. Every exposed user appears, even with no qualifying events; unexposed users are dropped. Return sorted [user, total].","evaluation_group":"w2-experiment-statistics-exposure-window","failed_approach":"Excluding the exposure day itself drops the first, often strongest, day of impact.","family":"w2-experiment-statistics-exposure-window-pre-exposure-events","id":"FA-74541","implementations":{"attempt":{"sha256":"c97fda9386f6e30a6d99c8a7c280b93009afb90698c9f53ee99483e2fa03f9d6","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(events, exposures, window_days):\n    totals = {u: 0 for u in exposures}\n    for user, day, value in events:\n        if user not in exposures:\n            continue\n        start = exposures[user]\n        if start < day < start + window_days:\n            totals[user] += value\n    return sorted([u, t] for u, t in totals.items())\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),\n  ('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),\n  ('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),\n  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),\n  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),\n  ('event log sample 1',\n   [[['a', 7, 1], ['b', 7, 1], ['d', 10, 1], ['e', 10, 1], ['b', 5, 1], ['c', 11, 1]],\n    {'c': 5, 'a': 0, 'b': 4},\n    1],\n   [['a', 0], ['b', 0], ['c', 0]]),\n  ('event log sample 2',\n   [[['c', 1, 1], ['a', 6, 2], ['a', 2, 5]], {'c': 3, 'd': 3}, 7],\n   [['c', 0], ['d', 0]]),\n  ('event log sample 3',\n   [[['b', 11, 5], ['e', 0, 5], ['e', 7, 5], ['c', 6, 5]], {'b': 2, 'c': 2, 'a': 2}, 1],\n   [['a', 0], ['b', 0], ['c', 0]])],\n [('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),\n  ('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),\n  ('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),\n  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),\n  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),\n  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),\n  ('event log sample 24',\n   [[['c', 5, 5], ['c', 9, 2], ['c', 6, 5], ['c', 6, 5], ['b', 11, 1], ['e', 6, 5], ['c', 2, 2]],\n    {'c': 2, 'b': 3},\n    7],\n   [['b', 0], ['c', 17]]),\n  ('event log sample 44',\n   [[['b', 9, 5], ['b', 6, 2], ['e', 8, 1], ['b', 3, 2], ['a', 3, 5], ['a', 8, 1], ['e', 8, 1]],\n    {'b': 4, 'c': 0},\n    7],\n   [['b', 7], ['c', 0]])],\n [('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),\n  ('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),\n  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),\n  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),\n  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),\n  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),\n  ('event log sample 11', [[['c', 7, 2], ['b', 3, 1]], {'a': 4, 'd': 2}, 1], [['a', 0], ['d', 0]]),\n  ('event log sample 26',\n   [[['a', 2, 5], ['d', 4, 5], ['c', 4, 1], ['c', 1, 2], ['e', 3, 5], ['e', 9, 1], ['e', 9, 1]],\n    {'c': 5, 'a': 5, 'b': 3},\n    3],\n   [['a', 0], ['b', 0], ['c', 0]])],\n [('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),\n  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),\n  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),\n  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),\n  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),\n  ('event log sample 7',\n   [[['d', 4, 1], ['a', 5, 2], ['a', 10, 5], ['b', 3, 2], ['a', 3, 2], ['c', 0, 1]], {'b': 5, 'a': 5}, 3],\n   [['a', 2], ['b', 0]]),\n  ('event log sample 16',\n   [[['d', 0, 5], ['b', 3, 2], ['d', 1, 1], ['d', 2, 1]], {'c': 1, 'b': 5}, 1],\n   [['b', 0], ['c', 0]]),\n  ('event log sample 24',\n   [[['c', 5, 5], ['c', 9, 2], ['c', 6, 5], ['c', 6, 5], ['b', 11, 1], ['e', 6, 5], ['c', 2, 2]],\n    {'c': 2, 'b': 3},\n    7],\n   [['b', 0], ['c', 17]])],\n [('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),\n  ('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),\n  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),\n  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),\n  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),\n  ('event log sample 2',\n   [[['c', 1, 1], ['a', 6, 2], ['a', 2, 5]], {'c': 3, 'd': 3}, 7],\n   [['c', 0], ['d', 0]]),\n  ('event log sample 21', [[], {'d': 5, 'c': 2}, 7], [['c', 0], ['d', 0]]),\n  ('event log sample 22', [[['e', 3, 2]], {'c': 1, 'b': 4}, 1], [['b', 0], ['c', 0]])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"broken":{"sha256":"6be0ab246064b77b00bab99f598f83b238d6f3c878a754255663a65a75cead12","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(events, exposures, window_days):\n    totals = {u: 0 for u in exposures}\n    for user, day, value in events:\n        if user not in exposures:\n            continue\n        start = exposures[user]\n        if day < start + window_days:\n            totals[user] += value\n    return sorted([u, t] for u, t in totals.items())\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),\n  ('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),\n  ('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),\n  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),\n  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),\n  ('event log sample 1',\n   [[['a', 7, 1], ['b', 7, 1], ['d', 10, 1], ['e', 10, 1], ['b', 5, 1], ['c', 11, 1]],\n    {'c': 5, 'a': 0, 'b': 4},\n    1],\n   [['a', 0], ['b', 0], ['c', 0]]),\n  ('event log sample 2',\n   [[['c', 1, 1], ['a', 6, 2], ['a', 2, 5]], {'c': 3, 'd': 3}, 7],\n   [['c', 0], ['d', 0]]),\n  ('event log sample 3',\n   [[['b', 11, 5], ['e', 0, 5], ['e', 7, 5], ['c', 6, 5]], {'b': 2, 'c': 2, 'a': 2}, 1],\n   [['a', 0], ['b', 0], ['c', 0]])],\n [('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),\n  ('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),\n  ('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),\n  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),\n  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),\n  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),\n  ('event log sample 24',\n   [[['c', 5, 5], ['c', 9, 2], ['c', 6, 5], ['c', 6, 5], ['b', 11, 1], ['e', 6, 5], ['c', 2, 2]],\n    {'c': 2, 'b': 3},\n    7],\n   [['b', 0], ['c', 17]]),\n  ('event log sample 44',\n   [[['b', 9, 5], ['b', 6, 2], ['e', 8, 1], ['b', 3, 2], ['a', 3, 5], ['a', 8, 1], ['e', 8, 1]],\n    {'b': 4, 'c': 0},\n    7],\n   [['b', 7], ['c', 0]])],\n [('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),\n  ('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),\n  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),\n  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),\n  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),\n  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),\n  ('event log sample 11', [[['c', 7, 2], ['b', 3, 1]], {'a': 4, 'd': 2}, 1], [['a', 0], ['d', 0]]),\n  ('event log sample 26',\n   [[['a', 2, 5], ['d', 4, 5], ['c', 4, 1], ['c', 1, 2], ['e', 3, 5], ['e', 9, 1], ['e', 9, 1]],\n    {'c': 5, 'a': 5, 'b': 3},\n    3],\n   [['a', 0], ['b', 0], ['c', 0]])],\n [('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),\n  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),\n  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),\n  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),\n  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),\n  ('event log sample 7',\n   [[['d', 4, 1], ['a', 5, 2], ['a', 10, 5], ['b', 3, 2], ['a', 3, 2], ['c', 0, 1]], {'b': 5, 'a': 5}, 3],\n   [['a', 2], ['b', 0]]),\n  ('event log sample 16',\n   [[['d', 0, 5], ['b', 3, 2], ['d', 1, 1], ['d', 2, 1]], {'c': 1, 'b': 5}, 1],\n   [['b', 0], ['c', 0]]),\n  ('event log sample 24',\n   [[['c', 5, 5], ['c', 9, 2], ['c', 6, 5], ['c', 6, 5], ['b', 11, 1], ['e', 6, 5], ['c', 2, 2]],\n    {'c': 2, 'b': 3},\n    7],\n   [['b', 0], ['c', 17]])],\n [('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),\n  ('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),\n  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),\n  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),\n  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),\n  ('event log sample 2',\n   [[['c', 1, 1], ['a', 6, 2], ['a', 2, 5]], {'c': 3, 'd': 3}, 7],\n   [['c', 0], ['d', 0]]),\n  ('event log sample 21', [[], {'d': 5, 'c': 2}, 7], [['c', 0], ['d', 0]]),\n  ('event log sample 22', [[['e', 3, 2]], {'c': 1, 'b': 4}, 1], [['b', 0], ['c', 0]])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"fixed":{"sha256":"b77bebfd1da0adca5225d12afcaf151c89045c098c2ac04e215ca438d28d428d","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(events, exposures, window_days):\n    totals = {u: 0 for u in exposures}\n    for user, day, value in events:\n        if user not in exposures:\n            continue\n        start = exposures[user]\n        if start <= day < start + window_days:\n            totals[user] += value\n    return sorted([u, t] for u, t in totals.items())\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),\n  ('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),\n  ('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),\n  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),\n  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),\n  ('event log sample 1',\n   [[['a', 7, 1], ['b', 7, 1], ['d', 10, 1], ['e', 10, 1], ['b', 5, 1], ['c', 11, 1]],\n    {'c': 5, 'a': 0, 'b': 4},\n    1],\n   [['a', 0], ['b', 0], ['c', 0]]),\n  ('event log sample 2',\n   [[['c', 1, 1], ['a', 6, 2], ['a', 2, 5]], {'c': 3, 'd': 3}, 7],\n   [['c', 0], ['d', 0]]),\n  ('event log sample 3',\n   [[['b', 11, 5], ['e', 0, 5], ['e', 7, 5], ['c', 6, 5]], {'b': 2, 'c': 2, 'a': 2}, 1],\n   [['a', 0], ['b', 0], ['c', 0]])],\n [('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),\n  ('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),\n  ('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),\n  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),\n  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),\n  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),\n  ('event log sample 24',\n   [[['c', 5, 5], ['c', 9, 2], ['c', 6, 5], ['c', 6, 5], ['b', 11, 1], ['e', 6, 5], ['c', 2, 2]],\n    {'c': 2, 'b': 3},\n    7],\n   [['b', 0], ['c', 17]]),\n  ('event log sample 44',\n   [[['b', 9, 5], ['b', 6, 2], ['e', 8, 1], ['b', 3, 2], ['a', 3, 5], ['a', 8, 1], ['e', 8, 1]],\n    {'b': 4, 'c': 0},\n    7],\n   [['b', 7], ['c', 0]])],\n [('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),\n  ('window end is exclusive', [[['a', 3, 1], ['a', 10, 9]], {'a': 3}, 7], [['a', 1]]),\n  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),\n  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),\n  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),\n  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),\n  ('event log sample 11', [[['c', 7, 2], ['b', 3, 1]], {'a': 4, 'd': 2}, 1], [['a', 0], ['d', 0]]),\n  ('event log sample 26',\n   [[['a', 2, 5], ['d', 4, 5], ['c', 4, 1], ['c', 1, 2], ['e', 3, 5], ['e', 9, 1], ['e', 9, 1]],\n    {'c': 5, 'a': 5, 'b': 3},\n    3],\n   [['a', 0], ['b', 0], ['c', 0]])],\n [('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),\n  ('last day inside the window counts', [[['b', 9, 2]], {'b': 3}, 7], [['b', 2]]),\n  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),\n  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),\n  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),\n  ('event log sample 7',\n   [[['d', 4, 1], ['a', 5, 2], ['a', 10, 5], ['b', 3, 2], ['a', 3, 2], ['c', 0, 1]], {'b': 5, 'a': 5}, 3],\n   [['a', 2], ['b', 0]]),\n  ('event log sample 16',\n   [[['d', 0, 5], ['b', 3, 2], ['d', 1, 1], ['d', 2, 1]], {'c': 1, 'b': 5}, 1],\n   [['b', 0], ['c', 0]]),\n  ('event log sample 24',\n   [[['c', 5, 5], ['c', 9, 2], ['c', 6, 5], ['c', 6, 5], ['b', 11, 1], ['e', 6, 5], ['c', 2, 2]],\n    {'c': 2, 'b': 3},\n    7],\n   [['b', 0], ['c', 17]])],\n [('pre-exposure activity is excluded', [[['a', 1, 5], ['a', 3, 2]], {'a': 3}, 7], [['a', 2]]),\n  ('exposure day itself counts', [[['a', 3, 4]], {'a': 3}, 7], [['a', 4]]),\n  ('exposed user with no events reports zero', [[['a', 3, 1]], {'a': 3, 'b': 4}, 7], [['a', 1], ['b', 0]]),\n  ('each user has their own window', [[['a', 2, 1], ['b', 8, 1]], {'a': 0, 'b': 6}, 3], [['a', 1], ['b', 1]]),\n  ('unexposed users are dropped', [[['z', 1, 9]], {'a': 0}, 3], [['a', 0]]),\n  ('event log sample 2',\n   [[['c', 1, 1], ['a', 6, 2], ['a', 2, 5]], {'c': 3, 'd': 3}, 7],\n   [['c', 0], ['d', 0]]),\n  ('event log sample 21', [[], {'d': 5, 'c': 2}, 7], [['c', 0], ['d', 0]]),\n  ('event log sample 22', [[['e', 3, 2]], {'c': 1, 'b': 4}, 1], [['b', 0], ['c', 0]])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"}},"limitations":"A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.","method":"Deterministic executable model with adversarial boundary fixtures.","provenance":{"created_by":"Failure Map","dependencies":"Python standard library","family":"w2-experiment-statistics-exposure-window-pre-exposure-events","generated_at":"2026-09-29T14:48:57.814810+00:00","license":"CC0-1.0","python":"3.12.14","seed":1,"split":"open-access"},"relevance":"Counting pre-exposure activity dilutes effects; dropping silent users biases means upward.","repair":"Require day >= exposure day.","root_cause":"The window check only bounds the end, not the exposure start.","sha256":"f2ce8cb92fe0074b60c8a8f17d781b497e14daede09b51a52e1c2ec2e7f44f1c","title":"Post-exposure metric window: Activity before exposure is counted · case 01","variant":1,"variant_policy":"Five numbered records share a model and may reuse boundary fixtures.","verification":{"attempt":{"elapsed_ms":37.138,"exit_code":1,"observations":[{"actual":[["a",0]],"check":"pre-exposure activity is excluded","expected":[["a",2]],"passed":false},{"actual":[["a",0]],"check":"exposure day itself counts","expected":[["a",4]],"passed":false},{"actual":[["a",0]],"check":"window end is exclusive","expected":[["a",1]],"passed":false},{"actual":[["b",2]],"check":"last day inside the window counts","expected":[["b",2]],"passed":true},{"actual":[["a",0],["b",0]],"check":"exposed user with no events reports zero","expected":[["a",1],["b",0]],"passed":false},{"actual":[["a",0],["b",0],["c",0]],"check":"event log sample 1","expected":[["a",0],["b",0],["c",0]],"passed":true},{"actual":[["c",0],["d",0]],"check":"event log sample 2","expected":[["c",0],["d",0]],"passed":true},{"actual":[["a",0],["b",0],["c",0]],"check":"event log sample 3","expected":[["a",0],["b",0],["c",0]],"passed":true}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"pre-exposure activity is excluded\", \"actual\": [[\"a\", 0]], \"expected\": [[\"a\", 2]], \"passed\": false}, {\"check\": \"exposure day itself counts\", \"actual\": [[\"a\", 0]], \"expected\": [[\"a\", 4]], \"passed\": false}, {\"check\": \"window end is exclusive\", \"actual\": [[\"a\", 0]], \"expected\": [[\"a\", 1]], \"passed\": false}, {\"check\": \"last day inside the window counts\", \"actual\": [[\"b\", 2]], \"expected\": [[\"b\", 2]], \"passed\": true}, {\"check\": \"exposed user with no events reports zero\", \"actual\": [[\"a\", 0], [\"b\", 0]], \"expected\": [[\"a\", 1], [\"b\", 0]], \"passed\": false}, {\"check\": \"event log sample 1\", \"actual\": [[\"a\", 0], [\"b\", 0], [\"c\", 0]], \"expected\": [[\"a\", 0], [\"b\", 0], [\"c\", 0]], \"passed\": true}, {\"check\": \"event log sample 2\", \"actual\": [[\"c\", 0], [\"d\", 0]], \"expected\": [[\"c\", 0], [\"d\", 0]], \"passed\": true}, {\"check\": \"event log sample 3\", \"actual\": [[\"a\", 0], [\"b\", 0], [\"c\", 0]], \"expected\": [[\"a\", 0], [\"b\", 0], [\"c\", 0]], \"passed\": true}], \"passed\": false}\n"},"broken":{"elapsed_ms":38.081,"exit_code":1,"observations":[{"actual":[["a",7]],"check":"pre-exposure activity is excluded","expected":[["a",2]],"passed":false},{"actual":[["a",4]],"check":"exposure day itself counts","expected":[["a",4]],"passed":true},{"actual":[["a",1]],"check":"window end is exclusive","expected":[["a",1]],"passed":true},{"actual":[["b",2]],"check":"last day inside the window counts","expected":[["b",2]],"passed":true},{"actual":[["a",1],["b",0]],"check":"exposed user with no events reports zero","expected":[["a",1],["b",0]],"passed":true},{"actual":[["a",0],["b",0],["c",0]],"check":"event log sample 1","expected":[["a",0],["b",0],["c",0]],"passed":true},{"actual":[["c",1],["d",0]],"check":"event log sample 2","expected":[["c",0],["d",0]],"passed":false},{"actual":[["a",0],["b",0],["c",0]],"check":"event log sample 3","expected":[["a",0],["b",0],["c",0]],"passed":true}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"pre-exposure activity is excluded\", \"actual\": [[\"a\", 7]], \"expected\": [[\"a\", 2]], \"passed\": false}, {\"check\": \"exposure day itself counts\", \"actual\": [[\"a\", 4]], \"expected\": [[\"a\", 4]], \"passed\": true}, {\"check\": \"window end is exclusive\", \"actual\": [[\"a\", 1]], \"expected\": [[\"a\", 1]], \"passed\": true}, {\"check\": \"last day inside the window counts\", \"actual\": [[\"b\", 2]], \"expected\": [[\"b\", 2]], \"passed\": true}, {\"check\": \"exposed user with no events reports zero\", \"actual\": [[\"a\", 1], [\"b\", 0]], \"expected\": [[\"a\", 1], [\"b\", 0]], \"passed\": true}, {\"check\": \"event log sample 1\", \"actual\": [[\"a\", 0], [\"b\", 0], [\"c\", 0]], \"expected\": [[\"a\", 0], [\"b\", 0], [\"c\", 0]], \"passed\": true}, {\"check\": \"event log sample 2\", \"actual\": [[\"c\", 1], [\"d\", 0]], \"expected\": [[\"c\", 0], [\"d\", 0]], \"passed\": false}, {\"check\": \"event log sample 3\", \"actual\": [[\"a\", 0], [\"b\", 0], [\"c\", 0]], \"expected\": [[\"a\", 0], [\"b\", 0], [\"c\", 0]], \"passed\": true}], \"passed\": false}\n"},"fixed":{"elapsed_ms":41.468,"exit_code":0,"observations":[{"actual":[["a",2]],"check":"pre-exposure activity is excluded","expected":[["a",2]],"passed":true},{"actual":[["a",4]],"check":"exposure day itself counts","expected":[["a",4]],"passed":true},{"actual":[["a",1]],"check":"window end is exclusive","expected":[["a",1]],"passed":true},{"actual":[["b",2]],"check":"last day inside the window counts","expected":[["b",2]],"passed":true},{"actual":[["a",1],["b",0]],"check":"exposed user with no events reports zero","expected":[["a",1],["b",0]],"passed":true},{"actual":[["a",0],["b",0],["c",0]],"check":"event log sample 1","expected":[["a",0],["b",0],["c",0]],"passed":true},{"actual":[["c",0],["d",0]],"check":"event log sample 2","expected":[["c",0],["d",0]],"passed":true},{"actual":[["a",0],["b",0],["c",0]],"check":"event log sample 3","expected":[["a",0],["b",0],["c",0]],"passed":true}],"passed":true,"stderr":"","stdout":"{\"observations\": [{\"check\": \"pre-exposure activity is excluded\", \"actual\": [[\"a\", 2]], \"expected\": [[\"a\", 2]], \"passed\": true}, {\"check\": \"exposure day itself counts\", \"actual\": [[\"a\", 4]], \"expected\": [[\"a\", 4]], \"passed\": true}, {\"check\": \"window end is exclusive\", \"actual\": [[\"a\", 1]], \"expected\": [[\"a\", 1]], \"passed\": true}, {\"check\": \"last day inside the window counts\", \"actual\": [[\"b\", 2]], \"expected\": [[\"b\", 2]], \"passed\": true}, {\"check\": \"exposed user with no events reports zero\", \"actual\": [[\"a\", 1], [\"b\", 0]], \"expected\": [[\"a\", 1], [\"b\", 0]], \"passed\": true}, {\"check\": \"event log sample 1\", \"actual\": [[\"a\", 0], [\"b\", 0], [\"c\", 0]], \"expected\": [[\"a\", 0], [\"b\", 0], [\"c\", 0]], \"passed\": true}, {\"check\": \"event log sample 2\", \"actual\": [[\"c\", 0], [\"d\", 0]], \"expected\": [[\"c\", 0], [\"d\", 0]], \"passed\": true}, {\"check\": \"event log sample 3\", \"actual\": [[\"a\", 0], [\"b\", 0], [\"c\", 0]], \"expected\": [[\"a\", 0], [\"b\", 0], [\"c\", 0]], \"passed\": true}], \"passed\": true}\n"}},"verified":true,"visibility":"public"}