{"abstract":"A 5 percent holdout actually withholds 6 percent of users, and 0 percent still withholds bucket 0.","category":"Feature flag rollout bucketing","checks":8,"contract":"h = crc32(\"holdout:\" + user) % 100 and b = crc32(flag + \":\" + user) % 100. Users listed exactly in force get treatment (forced). Otherwise h < holdout_pct puts the user in the global holdout (control, holdout). Otherwise b < rollout gives treatment, else control (reason rollout). Return [variant, reason, h, b].","contract_signature":"user, flag, holdout_pct, rollout, force","evaluation_group":"w2-feature-flag-rollout-bucketing-global-holdout","failed_approach":"Subtracting one from the limit removes the last legitimate holdout bucket.","family":"w2-feature-flag-rollout-bucketing-global-holdout-holdout-boundary","id":"FA-74101","implementations":{"attempt":{"sha256":"f10989223be0fa4a0f6cc0d48e565ed4bcdda57fd33b4e6c002fda3e3704a21d","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\nimport zlib\nN = 1\nobservations = []\ndef solve(user, flag, holdout_pct, rollout, force):\n    h = zlib.crc32(('holdout:' + user).encode()) % 100\n    b = zlib.crc32((flag + ':' + user).encode()) % 100\n    if user in force:\n        return ['treatment', 'forced', h, b]\n    if h < holdout_pct - 1:\n        return ['control', 'holdout', h, b]\n    if b < rollout:\n        return ['treatment', 'rollout', h, b]\n    return ['control', 'rollout', h, b]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),\n  ('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),\n  ('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),\n  ('holdout user with full rollout stays control',\n   ['u14', 'checkout', 10, 100, []],\n   ['control', 'holdout', 9, 58]),\n  ('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),\n  ('holdout sample 1', ['u1665', 'search', 93, 0, ['qa']], ['control', 'holdout', 92, 6]),\n  ('holdout sample 2', ['u2377', 'search', 10, 20, ['u2']], ['control', 'rollout', 24, 94]),\n  ('holdout sample 3', ['u1179', 'checkout', 75, 20, []], ['control', 'rollout', 75, 54])],\n [('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),\n  ('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),\n  ('holdout user with full rollout stays control',\n   ['u14', 'checkout', 10, 100, []],\n   ['control', 'holdout', 9, 58]),\n  ('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),\n  ('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),\n  ('holdout sample 6', ['u1739', 'checkout', 74, 100, ['u1739']], ['treatment', 'forced', 73, 48]),\n  ('holdout sample 26', ['u2754', 'checkout', 20, 100, []], ['treatment', 'rollout', 20, 17]),\n  ('holdout sample 52', ['u2085', 'search', 55, 20, []], ['control', 'holdout', 54, 96])],\n [('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),\n  ('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),\n  ('holdout user with full rollout stays control',\n   ['u14', 'checkout', 10, 100, []],\n   ['control', 'holdout', 9, 58]),\n  ('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),\n  ('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),\n  ('holdout sample 11', ['u2412', 'checkout', 50, 50, []], ['control', 'rollout', 76, 65]),\n  ('holdout sample 40', ['u1223', 'checkout', 10, 20, ['u1']], ['control', 'holdout', 9, 8]),\n  ('holdout sample 51', ['u1347', 'search', 65, 50, ['qa']], ['control', 'rollout', 65, 59])],\n [('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),\n  ('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),\n  ('holdout user with full rollout stays control',\n   ['u14', 'checkout', 10, 100, []],\n   ['control', 'holdout', 9, 58]),\n  ('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),\n  ('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),\n  ('holdout sample 16', ['u1149', 'search', 48, 50, ['qa']], ['control', 'rollout', 48, 62]),\n  ('holdout sample 23', ['u1007', 'checkout', 52, 100, ['qa']], ['treatment', 'rollout', 52, 1]),\n  ('holdout sample 31', ['u274', 'checkout', 96, 100, ['qa']], ['control', 'holdout', 95, 82])],\n [('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),\n  ('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),\n  ('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),\n  ('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),\n  ('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),\n  ('holdout sample 21', ['u1207', 'search', 0, 20, []], ['control', 'rollout', 66, 64]),\n  ('holdout sample 22', ['u2600', 'checkout', 88, 50, []], ['control', 'holdout', 87, 42]),\n  ('holdout sample 42', ['u309', 'search', 78, 20, ['u3']], ['control', 'rollout', 78, 24])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"broken":{"sha256":"3fd9c9c67adb744db41bff5858164f7fc1ef27c6dd9cdfe17c724ab60ec04b83","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\nimport zlib\nN = 1\nobservations = []\ndef solve(user, flag, holdout_pct, rollout, force):\n    h = zlib.crc32(('holdout:' + user).encode()) % 100\n    b = zlib.crc32((flag + ':' + user).encode()) % 100\n    if user in force:\n        return ['treatment', 'forced', h, b]\n    if h <= holdout_pct:\n        return ['control', 'holdout', h, b]\n    if b < rollout:\n        return ['treatment', 'rollout', h, b]\n    return ['control', 'rollout', h, b]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),\n  ('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),\n  ('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),\n  ('holdout user with full rollout stays control',\n   ['u14', 'checkout', 10, 100, []],\n   ['control', 'holdout', 9, 58]),\n  ('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),\n  ('holdout sample 1', ['u1665', 'search', 93, 0, ['qa']], ['control', 'holdout', 92, 6]),\n  ('holdout sample 2', ['u2377', 'search', 10, 20, ['u2']], ['control', 'rollout', 24, 94]),\n  ('holdout sample 3', ['u1179', 'checkout', 75, 20, []], ['control', 'rollout', 75, 54])],\n [('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),\n  ('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),\n  ('holdout user with full rollout stays control',\n   ['u14', 'checkout', 10, 100, []],\n   ['control', 'holdout', 9, 58]),\n  ('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),\n  ('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),\n  ('holdout sample 6', ['u1739', 'checkout', 74, 100, ['u1739']], ['treatment', 'forced', 73, 48]),\n  ('holdout sample 26', ['u2754', 'checkout', 20, 100, []], ['treatment', 'rollout', 20, 17]),\n  ('holdout sample 52', ['u2085', 'search', 55, 20, []], ['control', 'holdout', 54, 96])],\n [('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),\n  ('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),\n  ('holdout user with full rollout stays control',\n   ['u14', 'checkout', 10, 100, []],\n   ['control', 'holdout', 9, 58]),\n  ('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),\n  ('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),\n  ('holdout sample 11', ['u2412', 'checkout', 50, 50, []], ['control', 'rollout', 76, 65]),\n  ('holdout sample 40', ['u1223', 'checkout', 10, 20, ['u1']], ['control', 'holdout', 9, 8]),\n  ('holdout sample 51', ['u1347', 'search', 65, 50, ['qa']], ['control', 'rollout', 65, 59])],\n [('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),\n  ('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),\n  ('holdout user with full rollout stays control',\n   ['u14', 'checkout', 10, 100, []],\n   ['control', 'holdout', 9, 58]),\n  ('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),\n  ('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),\n  ('holdout sample 16', ['u1149', 'search', 48, 50, ['qa']], ['control', 'rollout', 48, 62]),\n  ('holdout sample 23', ['u1007', 'checkout', 52, 100, ['qa']], ['treatment', 'rollout', 52, 1]),\n  ('holdout sample 31', ['u274', 'checkout', 96, 100, ['qa']], ['control', 'holdout', 95, 82])],\n [('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),\n  ('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),\n  ('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),\n  ('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),\n  ('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),\n  ('holdout sample 21', ['u1207', 'search', 0, 20, []], ['control', 'rollout', 66, 64]),\n  ('holdout sample 22', ['u2600', 'checkout', 88, 50, []], ['control', 'holdout', 87, 42]),\n  ('holdout sample 42', ['u309', 'search', 78, 20, ['u3']], ['control', 'rollout', 78, 24])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"}},"limitations":"A deterministic toy flag-evaluation model with a stipulated contract; it does not reproduce any vendor SDK byte for byte. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.","method":"Deterministic executable model with adversarial boundary fixtures.","provenance":{"created_by":"Failure Map","dependencies":"Python standard library","family":"w2-feature-flag-rollout-bucketing-global-holdout-holdout-boundary","generated_at":"2026-09-29T14:48:53.623292+00:00","license":"CC0-1.0","python":"3.12.14","seed":1,"split":"open-access"},"relevance":"A global holdout measures the cumulative effect of all launches and must stay uncontaminated.","root_cause":"Holdout membership uses h <= holdout_pct.","sha256":"0fc5d169f8a1e903182c9e9ae8aaf21f942b976274bb4ece2652d3fb742d8fda","title":"Global holdout exclusion: Holdout percentage admits one extra bucket · case 01","variant":1,"variant_policy":"Five numbered records share a model and may reuse boundary fixtures.","verified":true,"visibility":"public","verification":{"attempt":{"elapsed_ms":38.708,"exit_code":1,"observations":[{"actual":["treatment","forced",6,94],"check":"forced QA user inside the holdout","expected":["treatment","forced",6,94],"passed":true},{"actual":["treatment","rollout",9,44],"check":"forcing matches whole user ids only","expected":["control","holdout",9,44],"passed":false},{"actual":["treatment","rollout",6,94],"check":"holdout boundary is exclusive","expected":["treatment","rollout",6,94],"passed":true},{"actual":["treatment","rollout",9,58],"check":"holdout user with full rollout stays control","expected":["control","holdout",9,58],"passed":false},{"actual":["treatment","rollout",30,52],"check":"user outside holdout follows rollout","expected":["treatment","rollout",30,52],"passed":true},{"actual":["control","rollout",92,6],"check":"holdout sample 1","expected":["control","holdout",92,6],"passed":false},{"actual":["control","rollout",24,94],"check":"holdout sample 2","expected":["control","rollout",24,94],"passed":true},{"actual":["control","rollout",75,54],"check":"holdout sample 3","expected":["control","rollout",75,54],"passed":true}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"forced QA user inside the holdout\", \"actual\": [\"treatment\", \"forced\", 6, 94], \"expected\": [\"treatment\", \"forced\", 6, 94], \"passed\": true}, {\"check\": \"forcing matches whole user ids only\", \"actual\": [\"treatment\", \"rollout\", 9, 44], \"expected\": [\"control\", \"holdout\", 9, 44], \"passed\": false}, {\"check\": \"holdout boundary is exclusive\", \"actual\": [\"treatment\", \"rollout\", 6, 94], \"expected\": [\"treatment\", \"rollout\", 6, 94], \"passed\": true}, {\"check\": \"holdout user with full rollout stays control\", \"actual\": [\"treatment\", \"rollout\", 9, 58], \"expected\": [\"control\", \"holdout\", 9, 58], \"passed\": false}, {\"check\": \"user outside holdout follows rollout\", \"actual\": [\"treatment\", \"rollout\", 30, 52], \"expected\": [\"treatment\", \"rollout\", 30, 52], \"passed\": true}, {\"check\": \"holdout sample 1\", \"actual\": [\"control\", \"rollout\", 92, 6], \"expected\": [\"control\", \"holdout\", 92, 6], \"passed\": false}, {\"check\": \"holdout sample 2\", \"actual\": [\"control\", \"rollout\", 24, 94], \"expected\": [\"control\", \"rollout\", 24, 94], \"passed\": true}, {\"check\": \"holdout sample 3\", \"actual\": [\"control\", \"rollout\", 75, 54], \"expected\": [\"control\", \"rollout\", 75, 54], \"passed\": true}], \"passed\": false}\n"},"broken":{"elapsed_ms":37.84,"exit_code":1,"observations":[{"actual":["treatment","forced",6,94],"check":"forced QA user inside the holdout","expected":["treatment","forced",6,94],"passed":true},{"actual":["control","holdout",9,44],"check":"forcing matches whole user ids only","expected":["control","holdout",9,44],"passed":true},{"actual":["control","holdout",6,94],"check":"holdout boundary is exclusive","expected":["treatment","rollout",6,94],"passed":false},{"actual":["control","holdout",9,58],"check":"holdout user with full rollout stays control","expected":["control","holdout",9,58],"passed":true},{"actual":["treatment","rollout",30,52],"check":"user outside holdout follows rollout","expected":["treatment","rollout",30,52],"passed":true},{"actual":["control","holdout",92,6],"check":"holdout sample 1","expected":["control","holdout",92,6],"passed":true},{"actual":["control","rollout",24,94],"check":"holdout sample 2","expected":["control","rollout",24,94],"passed":true},{"actual":["control","holdout",75,54],"check":"holdout sample 3","expected":["control","rollout",75,54],"passed":false}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"forced QA user inside the holdout\", \"actual\": [\"treatment\", \"forced\", 6, 94], \"expected\": [\"treatment\", \"forced\", 6, 94], \"passed\": true}, {\"check\": \"forcing matches whole user ids only\", \"actual\": [\"control\", \"holdout\", 9, 44], \"expected\": [\"control\", \"holdout\", 9, 44], \"passed\": true}, {\"check\": \"holdout boundary is exclusive\", \"actual\": [\"control\", \"holdout\", 6, 94], \"expected\": [\"treatment\", \"rollout\", 6, 94], \"passed\": false}, {\"check\": \"holdout user with full rollout stays control\", \"actual\": [\"control\", \"holdout\", 9, 58], \"expected\": [\"control\", \"holdout\", 9, 58], \"passed\": true}, {\"check\": \"user outside holdout follows rollout\", \"actual\": [\"treatment\", \"rollout\", 30, 52], \"expected\": [\"treatment\", \"rollout\", 30, 52], \"passed\": true}, {\"check\": \"holdout sample 1\", \"actual\": [\"control\", \"holdout\", 92, 6], \"expected\": [\"control\", \"holdout\", 92, 6], \"passed\": true}, {\"check\": \"holdout sample 2\", \"actual\": [\"control\", \"rollout\", 24, 94], \"expected\": [\"control\", \"rollout\", 24, 94], \"passed\": true}, {\"check\": \"holdout sample 3\", \"actual\": [\"control\", \"holdout\", 75, 54], \"expected\": [\"control\", \"rollout\", 75, 54], \"passed\": false}], \"passed\": false}\n"}},"member_only":{"stages":["fixed"],"fields":["implementations.fixed","verification.fixed","harness","repair"],"note":"The verified repair, its recorded checks, the repair description, and the scoring harness are available to members."}}