FA-74721 / Experiment statistics / Open access

Cluster randomisation design effect: Cluster size variance uses a sample divisor · case 01

Design effects for a fixed set of clusters are inflated.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The spread of cluster sizes divides by k - 1 instead of k.

VERIFIED REPAIR

Use the population variance of the observed cluster sizes.

Unsuccessful approach: Dividing by k + 1 understates the size spread.

Case contract

For clusters (e.g. companies) randomised as units: mean size m, population variance of sizes / m^2 = cv^2, design effect = 1 + ((cv^2 + 1) m - 1) icc. Effective sample = total units / design effect; clusters needed = ceil(n_required * design effect / m). No clusters or zero mean -> None. Return [round(deff, 6), round(effective n, 6), clusters needed].

Why this case matters

B2B experiments randomise by account; ignoring clustering makes results look far more certain.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(cluster_sizes, icc, n_required):
    k = len(cluster_sizes)
    if k == 0:
        return None
    total = sum(cluster_sizes)
    mbar = total / k
    if mbar == 0:
        return None
    var = sum((s - mbar) ** 2 for s in cluster_sizes) / (k - 1)
    cv2 = var / mbar ** 2
    deff = 1 + ((cv2 + 1) * mbar - 1) * icc
    return [round(deff, 6), round(total / deff, 6), math.ceil(n_required * deff / mbar)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('unequal cluster sizes inflate the design effect', [[5, 10, 20, 45], 0.05, 500], [2.54375, 31.449631, 64]),
  ('equal clusters', [[10, 10, 10], 0.1, 300], [1.9, 15.789474, 57]),
  ('zero icc means no inflation', [[3, 40], 0.0, 100], [1.0, 43.0, 5]),
  ('small clusters', [[2, 4], 0.3, 50], [1.7, 3.529412, 29]),
  ('cluster sample 1', [[40, 20, 40, 20, 20, 40], 0.01, 500], [1.323333, 136.020151, 23]),
  ('cluster sample 2', [[3, 10, 10, 20], 0.05, 1000], [1.65814, 25.932679, 155]),
  ('cluster sample 3', [[40, 10, 5], 0.01, 500], [1.303636, 42.189679, 36]),
  ('cluster sample 4', [[20, 10, 20, 5, 5, 3], 0.2, 100], [3.844444, 16.387283, 37])],
 [('unequal cluster sizes inflate the design effect', [[5, 10, 20, 45], 0.05, 500], [2.54375, 31.449631, 64]),
  ('equal clusters', [[10, 10, 10], 0.1, 300], [1.9, 15.789474, 57]),
  ('zero icc means no inflation', [[3, 40], 0.0, 100], [1.0, 43.0, 5]),
  ('small clusters', [[2, 4], 0.3, 50], [1.7, 3.529412, 29]),
  ('cluster sample 8', [[5, 5, 3, 5], 0.2, 1000], [1.733333, 10.384615, 386]),
  ('cluster sample 9', [[5, 5, 10, 40, 20, 20], 0.2, 100], [5.9, 16.949153, 36]),
  ('cluster sample 12', [[40, 5, 10, 3, 40, 3], 0.0, 100], [1.0, 101.0, 6]),
  ('cluster sample 13', [[3, 3], 0.2, 500], [1.4, 4.285714, 234])],
 [('unequal cluster sizes inflate the design effect', [[5, 10, 20, 45], 0.05, 500], [2.54375, 31.449631, 64]),
  ('equal clusters', [[10, 10, 10], 0.1, 300], [1.9, 15.789474, 57]),
  ('zero icc means no inflation', [[3, 40], 0.0, 100], [1.0, 43.0, 5]),
  ('small clusters', [[2, 4], 0.3, 50], [1.7, 3.529412, 29]),
  ('cluster sample 15', [[40, 20, 20, 10], 0.0, 100], [1.0, 90.0, 5]),
  ('cluster sample 16', [[3, 20, 10, 5, 10, 40], 0.0, 500], [1.0, 88.0, 35]),
  ('cluster sample 17', [[20, 40, 3, 20, 5], 0.05, 100], [2.332955, 37.720409, 14]),
  ('cluster sample 24', [[5, 10], 0.2, 100], [2.466667, 6.081081, 33])],
 [('unequal cluster sizes inflate the design effect', [[5, 10, 20, 45], 0.05, 500], [2.54375, 31.449631, 64]),
  ('equal clusters', [[10, 10, 10], 0.1, 300], [1.9, 15.789474, 57]),
  ('zero icc means no inflation', [[3, 40], 0.0, 100], [1.0, 43.0, 5]),
  ('small clusters', [[2, 4], 0.3, 50], [1.7, 3.529412, 29]),
  ('cluster sample 21', [[40, 10, 5], 0.05, 100], [2.518182, 21.841155, 14]),
  ('cluster sample 23', [[40, 5, 40], 0.01, 100], [1.369412, 62.070447, 5]),
  ('cluster sample 24', [[5, 10], 0.2, 100], [2.466667, 6.081081, 33]),
  ('cluster sample 39', [[5, 10], 0.01, 500], [1.073333, 13.975155, 72])],
 [('unequal cluster sizes inflate the design effect', [[5, 10, 20, 45], 0.05, 500], [2.54375, 31.449631, 64]),
  ('equal clusters', [[10, 10, 10], 0.1, 300], [1.9, 15.789474, 57]),
  ('zero icc means no inflation', [[3, 40], 0.0, 100], [1.0, 43.0, 5]),
  ('small clusters', [[2, 4], 0.3, 50], [1.7, 3.529412, 29]),
  ('cluster sample 29', [[3, 40, 40, 40], 0.01, 500], [1.380976, 89.067467, 23]),
  ('cluster sample 30', [[3, 5, 3, 20, 10], 0.0, 1000], [1.0, 41.0, 122]),
  ('cluster sample 31', [[10, 20, 5], 0.0, 1000], [1.0, 35.0, 86]),
  ('cluster sample 49', [[20, 10, 20, 20, 10, 3], 0.01, 1000], [1.159759, 71.56659, 84])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
unequal cluster sizes inflate the design effect	`[2.741667, 29.179331, 69]`	`[2.54375, 31.449631, 64]`	Failed
equal clusters	`[1.9, 15.789474, 57]`	`[1.9, 15.789474, 57]`	Passed
zero icc means no inflation	`[1.0, 43.0, 5]`	`[1.0, 43.0, 5]`	Passed
small clusters	`[1.8, 3.333333, 30]`	`[1.7, 3.529412, 29]`	Failed
cluster sample 1	`[1.33, 135.338346, 23]`	`[1.323333, 136.020151, 23]`	Failed
cluster sample 2	`[1.715019, 25.072603, 160]`	`[1.65814, 25.932679, 155]`	Failed
cluster sample 3	`[1.368788, 40.181536, 38]`	`[1.303636, 42.189679, 36]`	Failed
cluster sample 4	`[4.033333, 15.619835, 39]`	`[3.844444, 16.387283, 37]`	Failed

SHA-256 / d963d50b225d8bd439b3c36dc9e276e1695374fe6054ecebb8e09faf69a02a79

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(cluster_sizes, icc, n_required):
    k = len(cluster_sizes)
    if k == 0:
        return None
    total = sum(cluster_sizes)
    mbar = total / k
    if mbar == 0:
        return None
    var = sum((s - mbar) ** 2 for s in cluster_sizes) / (k + 1)
    cv2 = var / mbar ** 2
    deff = 1 + ((cv2 + 1) * mbar - 1) * icc
    return [round(deff, 6), round(total / deff, 6), math.ceil(n_required * deff / mbar)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('unequal cluster sizes inflate the design effect', [[5, 10, 20, 45], 0.05, 500], [2.54375, 31.449631, 64]),
  ('equal clusters', [[10, 10, 10], 0.1, 300], [1.9, 15.789474, 57]),
  ('zero icc means no inflation', [[3, 40], 0.0, 100], [1.0, 43.0, 5]),
  ('small clusters', [[2, 4], 0.3, 50], [1.7, 3.529412, 29]),
  ('cluster sample 1', [[40, 20, 40, 20, 20, 40], 0.01, 500], [1.323333, 136.020151, 23]),
  ('cluster sample 2', [[3, 10, 10, 20], 0.05, 1000], [1.65814, 25.932679, 155]),
  ('cluster sample 3', [[40, 10, 5], 0.01, 500], [1.303636, 42.189679, 36]),
  ('cluster sample 4', [[20, 10, 20, 5, 5, 3], 0.2, 100], [3.844444, 16.387283, 37])],
 [('unequal cluster sizes inflate the design effect', [[5, 10, 20, 45], 0.05, 500], [2.54375, 31.449631, 64]),
  ('equal clusters', [[10, 10, 10], 0.1, 300], [1.9, 15.789474, 57]),
  ('zero icc means no inflation', [[3, 40], 0.0, 100], [1.0, 43.0, 5]),
  ('small clusters', [[2, 4], 0.3, 50], [1.7, 3.529412, 29]),
  ('cluster sample 8', [[5, 5, 3, 5], 0.2, 1000], [1.733333, 10.384615, 386]),
  ('cluster sample 9', [[5, 5, 10, 40, 20, 20], 0.2, 100], [5.9, 16.949153, 36]),
  ('cluster sample 12', [[40, 5, 10, 3, 40, 3], 0.0, 100], [1.0, 101.0, 6]),
  ('cluster sample 13', [[3, 3], 0.2, 500], [1.4, 4.285714, 234])],
 [('unequal cluster sizes inflate the design effect', [[5, 10, 20, 45], 0.05, 500], [2.54375, 31.449631, 64]),
  ('equal clusters', [[10, 10, 10], 0.1, 300], [1.9, 15.789474, 57]),
  ('zero icc means no inflation', [[3, 40], 0.0, 100], [1.0, 43.0, 5]),
  ('small clusters', [[2, 4], 0.3, 50], [1.7, 3.529412, 29]),
  ('cluster sample 15', [[40, 20, 20, 10], 0.0, 100], [1.0, 90.0, 5]),
  ('cluster sample 16', [[3, 20, 10, 5, 10, 40], 0.0, 500], [1.0, 88.0, 35]),
  ('cluster sample 17', [[20, 40, 3, 20, 5], 0.05, 100], [2.332955, 37.720409, 14]),
  ('cluster sample 24', [[5, 10], 0.2, 100], [2.466667, 6.081081, 33])],
 [('unequal cluster sizes inflate the design effect', [[5, 10, 20, 45], 0.05, 500], [2.54375, 31.449631, 64]),
  ('equal clusters', [[10, 10, 10], 0.1, 300], [1.9, 15.789474, 57]),
  ('zero icc means no inflation', [[3, 40], 0.0, 100], [1.0, 43.0, 5]),
  ('small clusters', [[2, 4], 0.3, 50], [1.7, 3.529412, 29]),
  ('cluster sample 21', [[40, 10, 5], 0.05, 100], [2.518182, 21.841155, 14]),
  ('cluster sample 23', [[40, 5, 40], 0.01, 100], [1.369412, 62.070447, 5]),
  ('cluster sample 24', [[5, 10], 0.2, 100], [2.466667, 6.081081, 33]),
  ('cluster sample 39', [[5, 10], 0.01, 500], [1.073333, 13.975155, 72])],
 [('unequal cluster sizes inflate the design effect', [[5, 10, 20, 45], 0.05, 500], [2.54375, 31.449631, 64]),
  ('equal clusters', [[10, 10, 10], 0.1, 300], [1.9, 15.789474, 57]),
  ('zero icc means no inflation', [[3, 40], 0.0, 100], [1.0, 43.0, 5]),
  ('small clusters', [[2, 4], 0.3, 50], [1.7, 3.529412, 29]),
  ('cluster sample 29', [[3, 40, 40, 40], 0.01, 500], [1.380976, 89.067467, 23]),
  ('cluster sample 30', [[3, 5, 3, 20, 10], 0.0, 1000], [1.0, 41.0, 122]),
  ('cluster sample 31', [[10, 20, 5], 0.0, 1000], [1.0, 35.0, 86]),
  ('cluster sample 49', [[20, 10, 20, 20, 10, 3], 0.01, 1000], [1.159759, 71.56659, 84])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
unequal cluster sizes inflate the design effect	`[2.425, 32.989691, 61]`	`[2.54375, 31.449631, 64]`	Failed
equal clusters	`[1.9, 15.789474, 57]`	`[1.9, 15.789474, 57]`	Passed
zero icc means no inflation	`[1.0, 43.0, 5]`	`[1.0, 43.0, 5]`	Passed
small clusters	`[1.666667, 3.6, 28]`	`[1.7, 3.529412, 29]`	Failed
cluster sample 1	`[1.318571, 136.511376, 22]`	`[1.323333, 136.020151, 23]`	Failed
cluster sample 2	`[1.624012, 26.477643, 152]`	`[1.65814, 25.932679, 155]`	Failed
cluster sample 3	`[1.271061, 43.27095, 35]`	`[1.303636, 42.189679, 36]`	Failed
cluster sample 4	`[3.709524, 16.983312, 36]`	`[3.844444, 16.387283, 37]`	Failed

SHA-256 / 858ad3c6b988be7ab20f277381a5e206d6ff99c41362b2a335b477ed5fb490c8

3 / The verified repair

Exit 0

"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(cluster_sizes, icc, n_required):
    k = len(cluster_sizes)
    if k == 0:
        return None
    total = sum(cluster_sizes)
    mbar = total / k
    if mbar == 0:
        return None
    var = sum((s - mbar) ** 2 for s in cluster_sizes) / k
    cv2 = var / mbar ** 2
    deff = 1 + ((cv2 + 1) * mbar - 1) * icc
    return [round(deff, 6), round(total / deff, 6), math.ceil(n_required * deff / mbar)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('unequal cluster sizes inflate the design effect', [[5, 10, 20, 45], 0.05, 500], [2.54375, 31.449631, 64]),
  ('equal clusters', [[10, 10, 10], 0.1, 300], [1.9, 15.789474, 57]),
  ('zero icc means no inflation', [[3, 40], 0.0, 100], [1.0, 43.0, 5]),
  ('small clusters', [[2, 4], 0.3, 50], [1.7, 3.529412, 29]),
  ('cluster sample 1', [[40, 20, 40, 20, 20, 40], 0.01, 500], [1.323333, 136.020151, 23]),
  ('cluster sample 2', [[3, 10, 10, 20], 0.05, 1000], [1.65814, 25.932679, 155]),
  ('cluster sample 3', [[40, 10, 5], 0.01, 500], [1.303636, 42.189679, 36]),
  ('cluster sample 4', [[20, 10, 20, 5, 5, 3], 0.2, 100], [3.844444, 16.387283, 37])],
 [('unequal cluster sizes inflate the design effect', [[5, 10, 20, 45], 0.05, 500], [2.54375, 31.449631, 64]),
  ('equal clusters', [[10, 10, 10], 0.1, 300], [1.9, 15.789474, 57]),
  ('zero icc means no inflation', [[3, 40], 0.0, 100], [1.0, 43.0, 5]),
  ('small clusters', [[2, 4], 0.3, 50], [1.7, 3.529412, 29]),
  ('cluster sample 8', [[5, 5, 3, 5], 0.2, 1000], [1.733333, 10.384615, 386]),
  ('cluster sample 9', [[5, 5, 10, 40, 20, 20], 0.2, 100], [5.9, 16.949153, 36]),
  ('cluster sample 12', [[40, 5, 10, 3, 40, 3], 0.0, 100], [1.0, 101.0, 6]),
  ('cluster sample 13', [[3, 3], 0.2, 500], [1.4, 4.285714, 234])],
 [('unequal cluster sizes inflate the design effect', [[5, 10, 20, 45], 0.05, 500], [2.54375, 31.449631, 64]),
  ('equal clusters', [[10, 10, 10], 0.1, 300], [1.9, 15.789474, 57]),
  ('zero icc means no inflation', [[3, 40], 0.0, 100], [1.0, 43.0, 5]),
  ('small clusters', [[2, 4], 0.3, 50], [1.7, 3.529412, 29]),
  ('cluster sample 15', [[40, 20, 20, 10], 0.0, 100], [1.0, 90.0, 5]),
  ('cluster sample 16', [[3, 20, 10, 5, 10, 40], 0.0, 500], [1.0, 88.0, 35]),
  ('cluster sample 17', [[20, 40, 3, 20, 5], 0.05, 100], [2.332955, 37.720409, 14]),
  ('cluster sample 24', [[5, 10], 0.2, 100], [2.466667, 6.081081, 33])],
 [('unequal cluster sizes inflate the design effect', [[5, 10, 20, 45], 0.05, 500], [2.54375, 31.449631, 64]),
  ('equal clusters', [[10, 10, 10], 0.1, 300], [1.9, 15.789474, 57]),
  ('zero icc means no inflation', [[3, 40], 0.0, 100], [1.0, 43.0, 5]),
  ('small clusters', [[2, 4], 0.3, 50], [1.7, 3.529412, 29]),
  ('cluster sample 21', [[40, 10, 5], 0.05, 100], [2.518182, 21.841155, 14]),
  ('cluster sample 23', [[40, 5, 40], 0.01, 100], [1.369412, 62.070447, 5]),
  ('cluster sample 24', [[5, 10], 0.2, 100], [2.466667, 6.081081, 33]),
  ('cluster sample 39', [[5, 10], 0.01, 500], [1.073333, 13.975155, 72])],
 [('unequal cluster sizes inflate the design effect', [[5, 10, 20, 45], 0.05, 500], [2.54375, 31.449631, 64]),
  ('equal clusters', [[10, 10, 10], 0.1, 300], [1.9, 15.789474, 57]),
  ('zero icc means no inflation', [[3, 40], 0.0, 100], [1.0, 43.0, 5]),
  ('small clusters', [[2, 4], 0.3, 50], [1.7, 3.529412, 29]),
  ('cluster sample 29', [[3, 40, 40, 40], 0.01, 500], [1.380976, 89.067467, 23]),
  ('cluster sample 30', [[3, 5, 3, 20, 10], 0.0, 1000], [1.0, 41.0, 122]),
  ('cluster sample 31', [[10, 20, 5], 0.0, 1000], [1.0, 35.0, 86]),
  ('cluster sample 49', [[20, 10, 20, 20, 10, 3], 0.01, 1000], [1.159759, 71.56659, 84])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
unequal cluster sizes inflate the design effect	`[2.54375, 31.449631, 64]`	`[2.54375, 31.449631, 64]`	Passed
equal clusters	`[1.9, 15.789474, 57]`	`[1.9, 15.789474, 57]`	Passed
zero icc means no inflation	`[1.0, 43.0, 5]`	`[1.0, 43.0, 5]`	Passed
small clusters	`[1.7, 3.529412, 29]`	`[1.7, 3.529412, 29]`	Passed
cluster sample 1	`[1.323333, 136.020151, 23]`	`[1.323333, 136.020151, 23]`	Passed
cluster sample 2	`[1.65814, 25.932679, 155]`	`[1.65814, 25.932679, 155]`	Passed
cluster sample 3	`[1.303636, 42.189679, 36]`	`[1.303636, 42.189679, 36]`	Passed
cluster sample 4	`[3.844444, 16.387283, 37]`	`[3.844444, 16.387283, 37]`	Passed

SHA-256 / 97f8858a2d2b3732d1b5f11edb55ea0769406f5b930a70f52aa11045edff8516

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:59.416117+00:00.

Case digest / 77ebef9a05576843c21666ea66fb6b3b9804b8474276fde1caa617939c3e204c