{"abstract":"Holdout membership differs per flag, so every user is held out of some launches and exposed to others.","category":"Feature flag rollout bucketing","checks":8,"contract":"h = crc32(\"holdout:\" + user) % 100 and b = crc32(flag + \":\" + user) % 100. Users listed exactly in force get treatment (forced). Otherwise h < holdout_pct puts the user in the global holdout (control, holdout). Otherwise b < rollout gives treatment, else control (reason rollout). Return [variant, reason, h, b].","contract_signature":"user, flag, holdout_pct, rollout, force","evaluation_group":"w2-feature-flag-rollout-bucketing-global-holdout","failed_approach":"Dropping the colon from the holdout prefix changes the hash input and moves users between groups.","family":"w2-feature-flag-rollout-bucketing-global-holdout-holdout-hash-independence","id":"FA-74096","implementations":{"attempt":{"sha256":"54b96009f6f356fe8a12613772625d12d1662c97207c6d69d5b0a3f9e25588f1","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\nimport zlib\nN = 1\nobservations = []\ndef solve(user, flag, holdout_pct, rollout, force):\n    h = zlib.crc32(('holdout' + user).encode()) % 100\n    b = zlib.crc32((flag + ':' + user).encode()) % 100\n    if user in force:\n        return ['treatment', 'forced', h, b]\n    if h < holdout_pct:\n        return ['control', 'holdout', h, b]\n    if b < rollout:\n        return ['treatment', 'rollout', h, b]\n    return ['control', 'rollout', h, b]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),\n  ('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),\n  ('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),\n  ('holdout user with full rollout stays control',\n   ['u14', 'checkout', 10, 100, []],\n   ['control', 'holdout', 9, 58]),\n  ('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),\n  ('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),\n  ('holdout sample 1', ['u1665', 'search', 93, 0, ['qa']], ['control', 'holdout', 92, 6]),\n  ('holdout sample 2', ['u2377', 'search', 10, 20, ['u2']], ['control', 'rollout', 24, 94])],\n [('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),\n  ('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),\n  ('holdout user with full rollout stays control',\n   ['u14', 'checkout', 10, 100, []],\n   ['control', 'holdout', 9, 58]),\n  ('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),\n  ('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),\n  ('holdout sample 2', ['u2377', 'search', 10, 20, ['u2']], ['control', 'rollout', 24, 94]),\n  ('holdout sample 6', ['u1739', 'checkout', 74, 100, ['u1739']], ['treatment', 'forced', 73, 48]),\n  ('holdout sample 7', ['u2898', 'search', 11, 50, []], ['control', 'holdout', 10, 20])],\n [('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),\n  ('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),\n  ('holdout user with full rollout stays control',\n   ['u14', 'checkout', 10, 100, []],\n   ['control', 'holdout', 9, 58]),\n  ('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),\n  ('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),\n  ('holdout sample 9', ['u1996', 'search', 16, 20, ['u1']], ['control', 'rollout', 16, 78]),\n  ('holdout sample 11', ['u2412', 'checkout', 50, 50, []], ['control', 'rollout', 76, 65]),\n  ('holdout sample 12', ['u764', 'search', 69, 20, ['qa']], ['control', 'rollout', 69, 79])],\n [('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),\n  ('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),\n  ('holdout user with full rollout stays control',\n   ['u14', 'checkout', 10, 100, []],\n   ['control', 'holdout', 9, 58]),\n  ('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),\n  ('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),\n  ('holdout sample 16', ['u1149', 'search', 48, 50, ['qa']], ['control', 'rollout', 48, 62]),\n  ('holdout sample 17', ['u2253', 'search', 8, 100, ['qa']], ['treatment', 'rollout', 8, 26]),\n  ('holdout sample 18', ['u1517', 'checkout', 0, 20, ['qa']], ['control', 'rollout', 86, 51])],\n [('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),\n  ('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),\n  ('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),\n  ('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),\n  ('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),\n  ('holdout sample 21', ['u1207', 'search', 0, 20, []], ['control', 'rollout', 66, 64]),\n  ('holdout sample 22', ['u2600', 'checkout', 88, 50, []], ['control', 'holdout', 87, 42]),\n  ('holdout sample 23', ['u1007', 'checkout', 52, 100, ['qa']], ['treatment', 'rollout', 52, 1])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"broken":{"sha256":"66a4cfc4312aa66020f9f81d34b549b933d98c77e727a6d67d500e09317fe56a","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\nimport zlib\nN = 1\nobservations = []\ndef solve(user, flag, holdout_pct, rollout, force):\n    h = zlib.crc32((flag + ':' + user).encode()) % 100\n    b = zlib.crc32((flag + ':' + user).encode()) % 100\n    if user in force:\n        return ['treatment', 'forced', h, b]\n    if h < holdout_pct:\n        return ['control', 'holdout', h, b]\n    if b < rollout:\n        return ['treatment', 'rollout', h, b]\n    return ['control', 'rollout', h, b]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),\n  ('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),\n  ('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),\n  ('holdout user with full rollout stays control',\n   ['u14', 'checkout', 10, 100, []],\n   ['control', 'holdout', 9, 58]),\n  ('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),\n  ('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),\n  ('holdout sample 1', ['u1665', 'search', 93, 0, ['qa']], ['control', 'holdout', 92, 6]),\n  ('holdout sample 2', ['u2377', 'search', 10, 20, ['u2']], ['control', 'rollout', 24, 94])],\n [('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),\n  ('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),\n  ('holdout user with full rollout stays control',\n   ['u14', 'checkout', 10, 100, []],\n   ['control', 'holdout', 9, 58]),\n  ('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),\n  ('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),\n  ('holdout sample 2', ['u2377', 'search', 10, 20, ['u2']], ['control', 'rollout', 24, 94]),\n  ('holdout sample 6', ['u1739', 'checkout', 74, 100, ['u1739']], ['treatment', 'forced', 73, 48]),\n  ('holdout sample 7', ['u2898', 'search', 11, 50, []], ['control', 'holdout', 10, 20])],\n [('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),\n  ('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),\n  ('holdout user with full rollout stays control',\n   ['u14', 'checkout', 10, 100, []],\n   ['control', 'holdout', 9, 58]),\n  ('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),\n  ('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),\n  ('holdout sample 9', ['u1996', 'search', 16, 20, ['u1']], ['control', 'rollout', 16, 78]),\n  ('holdout sample 11', ['u2412', 'checkout', 50, 50, []], ['control', 'rollout', 76, 65]),\n  ('holdout sample 12', ['u764', 'search', 69, 20, ['qa']], ['control', 'rollout', 69, 79])],\n [('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),\n  ('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),\n  ('holdout user with full rollout stays control',\n   ['u14', 'checkout', 10, 100, []],\n   ['control', 'holdout', 9, 58]),\n  ('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),\n  ('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),\n  ('holdout sample 16', ['u1149', 'search', 48, 50, ['qa']], ['control', 'rollout', 48, 62]),\n  ('holdout sample 17', ['u2253', 'search', 8, 100, ['qa']], ['treatment', 'rollout', 8, 26]),\n  ('holdout sample 18', ['u1517', 'checkout', 0, 20, ['qa']], ['control', 'rollout', 86, 51])],\n [('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),\n  ('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),\n  ('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),\n  ('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),\n  ('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),\n  ('holdout sample 21', ['u1207', 'search', 0, 20, []], ['control', 'rollout', 66, 64]),\n  ('holdout sample 22', ['u2600', 'checkout', 88, 50, []], ['control', 'holdout', 87, 42]),\n  ('holdout sample 23', ['u1007', 'checkout', 52, 100, ['qa']], ['treatment', 'rollout', 52, 1])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"}},"limitations":"A deterministic toy flag-evaluation model with a stipulated contract; it does not reproduce any vendor SDK byte for byte. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.","method":"Deterministic executable model with adversarial boundary fixtures.","provenance":{"created_by":"Failure Map","dependencies":"Python standard library","family":"w2-feature-flag-rollout-bucketing-global-holdout-holdout-hash-independence","generated_at":"2026-09-29T14:48:53.541389+00:00","license":"CC0-1.0","python":"3.12.14","seed":1,"split":"open-access"},"relevance":"A global holdout measures the cumulative effect of all launches and must stay uncontaminated.","root_cause":"h is computed from flag + \":\" + user, the same input as the rollout bucket.","sha256":"d3bcd9b0492f09c0e43d79ef9da4086ccc31393cb831d6bff9783ec07ad3c8e3","title":"Global holdout exclusion: The holdout reuses the flag bucket · case 01","variant":1,"variant_policy":"Five numbered records share a model and may reuse boundary fixtures.","verified":true,"visibility":"public","verification":{"attempt":{"elapsed_ms":41.664,"exit_code":1,"observations":[{"actual":["treatment","forced",0,94],"check":"forced QA user inside the holdout","expected":["treatment","forced",6,94],"passed":false},{"actual":["control","holdout",2,44],"check":"forcing matches whole user ids only","expected":["control","holdout",9,44],"passed":false},{"actual":["control","holdout",0,94],"check":"holdout boundary is exclusive","expected":["treatment","rollout",6,94],"passed":false},{"actual":["control","holdout",2,58],"check":"holdout user with full rollout stays control","expected":["control","holdout",9,58],"passed":false},{"actual":["treatment","rollout",8,52],"check":"user outside holdout follows rollout","expected":["treatment","rollout",30,52],"passed":false},{"actual":["control","rollout",81,5],"check":"zero rollout control","expected":["control","rollout",35,5],"passed":false},{"actual":["control","holdout",22,6],"check":"holdout sample 1","expected":["control","holdout",92,6],"passed":false},{"actual":["control","rollout",90,94],"check":"holdout sample 2","expected":["control","rollout",24,94],"passed":false}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"forced QA user inside the holdout\", \"actual\": [\"treatment\", \"forced\", 0, 94], \"expected\": [\"treatment\", \"forced\", 6, 94], \"passed\": false}, {\"check\": \"forcing matches whole user ids only\", \"actual\": [\"control\", \"holdout\", 2, 44], \"expected\": [\"control\", \"holdout\", 9, 44], \"passed\": false}, {\"check\": \"holdout boundary is exclusive\", \"actual\": [\"control\", \"holdout\", 0, 94], \"expected\": [\"treatment\", \"rollout\", 6, 94], \"passed\": false}, {\"check\": \"holdout user with full rollout stays control\", \"actual\": [\"control\", \"holdout\", 2, 58], \"expected\": [\"control\", \"holdout\", 9, 58], \"passed\": false}, {\"check\": \"user outside holdout follows rollout\", \"actual\": [\"treatment\", \"rollout\", 8, 52], \"expected\": [\"treatment\", \"rollout\", 30, 52], \"passed\": false}, {\"check\": \"zero rollout control\", \"actual\": [\"control\", \"rollout\", 81, 5], \"expected\": [\"control\", \"rollout\", 35, 5], \"passed\": false}, {\"check\": \"holdout sample 1\", \"actual\": [\"control\", \"holdout\", 22, 6], \"expected\": [\"control\", \"holdout\", 92, 6], \"passed\": false}, {\"check\": \"holdout sample 2\", \"actual\": [\"control\", \"rollout\", 90, 94], \"expected\": [\"control\", \"rollout\", 24, 94], \"passed\": false}], \"passed\": false}\n"},"broken":{"elapsed_ms":40.816,"exit_code":1,"observations":[{"actual":["treatment","forced",94,94],"check":"forced QA user inside the holdout","expected":["treatment","forced",6,94],"passed":false},{"actual":["treatment","rollout",44,44],"check":"forcing matches whole user ids only","expected":["control","holdout",9,44],"passed":false},{"actual":["treatment","rollout",94,94],"check":"holdout boundary is exclusive","expected":["treatment","rollout",6,94],"passed":false},{"actual":["treatment","rollout",58,58],"check":"holdout user with full rollout stays control","expected":["control","holdout",9,58],"passed":false},{"actual":["treatment","rollout",52,52],"check":"user outside holdout follows rollout","expected":["treatment","rollout",30,52],"passed":false},{"actual":["control","rollout",5,5],"check":"zero rollout control","expected":["control","rollout",35,5],"passed":false},{"actual":["control","holdout",6,6],"check":"holdout sample 1","expected":["control","holdout",92,6],"passed":false},{"actual":["control","rollout",94,94],"check":"holdout sample 2","expected":["control","rollout",24,94],"passed":false}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"forced QA user inside the holdout\", \"actual\": [\"treatment\", \"forced\", 94, 94], \"expected\": [\"treatment\", \"forced\", 6, 94], \"passed\": false}, {\"check\": \"forcing matches whole user ids only\", \"actual\": [\"treatment\", \"rollout\", 44, 44], \"expected\": [\"control\", \"holdout\", 9, 44], \"passed\": false}, {\"check\": \"holdout boundary is exclusive\", \"actual\": [\"treatment\", \"rollout\", 94, 94], \"expected\": [\"treatment\", \"rollout\", 6, 94], \"passed\": false}, {\"check\": \"holdout user with full rollout stays control\", \"actual\": [\"treatment\", \"rollout\", 58, 58], \"expected\": [\"control\", \"holdout\", 9, 58], \"passed\": false}, {\"check\": \"user outside holdout follows rollout\", \"actual\": [\"treatment\", \"rollout\", 52, 52], \"expected\": [\"treatment\", \"rollout\", 30, 52], \"passed\": false}, {\"check\": \"zero rollout control\", \"actual\": [\"control\", \"rollout\", 5, 5], \"expected\": [\"control\", \"rollout\", 35, 5], \"passed\": false}, {\"check\": \"holdout sample 1\", \"actual\": [\"control\", \"holdout\", 6, 6], \"expected\": [\"control\", \"holdout\", 92, 6], \"passed\": false}, {\"check\": \"holdout sample 2\", \"actual\": [\"control\", \"rollout\", 94, 94], \"expected\": [\"control\", \"rollout\", 24, 94], \"passed\": false}], \"passed\": false}\n"}},"member_only":{"stages":["fixed"],"fields":["implementations.fixed","verification.fixed","harness","repair"],"note":"The verified repair, its recorded checks, the repair description, and the scoring harness are available to members."}}