Statistical Power for Closed-List Campaigns

AB Testing under Non-Compliance

Platform match rates and partial delivery dilute your experiment. This simulator quantifies the ITT effect, derives LATE via the Wald estimator, and computes required sample sizes adjusted for platform non-compliance.

Intent-to-Treat (ITT) Wald Estimator / LATE Inflation Factor Power Analysis Non-Compliance Adjustment Closed-List Retargeting
ITT

Intent-to-Treat (ITT)

The safe, unbiased anchor

ITT compares everyone assigned to treatment vs. control, regardless of whether they actually saw the ad. It is always unbiased because it preserves the original randomization. The tradeoff: it dilutes the real effect since many treatment users never received the ad.
Practical Example
You upload 50,000 users. Treatment group: 25,000. Control: 25,000. The control converts at 5.0%, treatment at 5.5%. ITT = 5.5% − 5.0% = 0.5 pp This 0.5 pp is the average effect across all 25,000, including those who never saw the ad.
LATE

Local Average Treatment Effect (LATE)

Effect on those who actually saw the ad

LATE (also called CACE) estimates the true causal effect on users who were actually exposed to the ad — the "compliers". It uses the Wald IV estimator to "undo" the dilution: LATE = ITT ÷ compliance LATE is always larger than ITT. The lower the compliance, the more LATE amplifies the signal — but also the uncertainty.
Practical Example
If match rate = 70% and delivery rate = 60%, compliance = 42%. Using the ITT of 0.5 pp: LATE = 0.5 pp ÷ 0.42 = 1.19 pp The ad caused a 1.19 pp lift among the 42% of users who actually received it. But the confidence interval will be ~2.4× wider than ITT's.
⚠️
Important Assumption
This simulator assumes that match rate and delivery rate introduce no selection bias — i.e., the users who are matched and delivered ads are a random subset of the treatment group. In practice, platforms often prioritize delivery to users more likely to engage, which violates the exclusion restriction and may bias the LATE estimate. Consider this when interpreting results, especially for high-targeting platforms.
Effective Compliance i
—
m × d = platform reach rate
Inflation Factor i
—
1 / (m×d)² extra sample needed
Required N (total) i
—
to detect observed ITT lift
MDE (ITT) i
—
MDE LATE: —
Achieved Power i
—
at your current N and ITT lift
LATE Estimate (Wald) i
—
ITT / (m × d) — effect on exposed users
Days to StatSig i
—
based on daily traffic
Non-Compliance Funnel — Treatment Group How the list shrinks at each platform layer
Click Calculate to view funnel
Power vs. List Size
MDE vs. List Size (ITT & LATE)
Conversion Rate Distributions (KDE)
Time Evolution to StatSig
Required N vs. Compliance Rate
Inflation Factor Heatmap (m × d)
Full Results Table All computed statistics
Click Calculate to see results
Statistical Formulas & Methodology
Compliance & LATE
compliance = m × d LATE = ITT / (m × d) (Wald estimator) SE(LATE) = SE(ITT) / (m × d)
Sample Size (total)
k = treatment fraction σ² = p₁(1-p₁) + p₂(1-p₂) N = (z_α + z_β)² × σ² × (1/k + 1/(1-k)) ─────────────────────────────── ITT_lift² (when k=0.5, 1/k+1/(1-k)=4)
MDE (given N)
n₁ = N×k, n₂ = N×(1-k) MDE_ITT = (z_α + z_β) × √(σ² × (1/n₁ + 1/n₂)) MDE_LATE = MDE_ITT / (m × d)
Achieved Power
SE_ITT = √(p₁(1-p₁)/n₂ + p₂(1-p₂)/n₁) z_eff = ITT / SE_ITT Power = Φ(z_eff - z_α) + Φ(-z_eff - z_α) (two-sided; second term usually negligible)
95% Confidence Intervals
SE_ITT = √(p₁(1-p₁)/n₁ + p₂(1-p₂)/n₂) CI_ITT = ITT ± z_{1-α/2} × SE_ITT CI_LATE = LATE ± z_{1-α/2} × SE_LATE
Test Statistic (Z-test)
p_pool = (x₁ + x₂) / (n₁ + n₂) Z = (p̂₂ - p̂₁) ───────────────────────────── √(p_pool×(1-p_pool)×(1/n₁+1/n₂)) p-value = 2×Φ(-|Z|) [two-sided]

References & Literature

Core papers and resources underlying this simulator's statistical framework.

Braun & Schwartz, 2025
Divergent delivery bias in platform AB tests
Journal of Marketing Research — Why platform-optimized delivery violates SUTVA and biases LATE
Imbens & Angrist, 1994
Identification and Estimation of Local Average Treatment Effects
Econometrica — Foundational IV/LATE identification theory
Spotify Engineering, 2023
Encouragement Designs and Instrumental Variables for A/B Testing
Practical implementation of LATE estimation in industrial experimentation
Bansak, 2020
powerLATE: Power Analysis for LATE
R package for power analysis under non-compliance — conservative bounds-based approach
Facure, 2022
Causal Inference for the Brave and True — Chapter 9
Non-compliance and LATE estimation — open-source Python textbook
Remerge, 2024
Incrementality Tests 101: ITT, PSA, Ghost Ads, Ghost Bids
Practical comparison of incrementality methods for mobile retargeting
Angrist & Pischke, 2009
Mostly Harmless Econometrics
Canonical reference for IV/2SLS and LATE estimation
Gerber & Green, 2012
Field Experiments: Design, Analysis, and Interpretation
ITT/LATE duality, one-sided non-compliance, exclusion restriction
X (Twitter) Engineering, 2016
Power, MDE, and Bucket Size Estimation in A/B Tests
Practical power analysis for online experiments at scale
Kellogg / Facebook, 2019
A Comparison of Approaches to Advertising Measurement
Academic comparison of RCT, observational, and quasi-experimental methods
DoorDash, 2024
Measuring Restaurant Ad Incrementality with Ghost Ads
Real-world application of Ghost Ads methodology
Kohavi et al., 2020
Trustworthy Online Controlled Experiments
Cambridge University Press — Practical guide to AB testing at scale