What this calculator tells you
An A/B test is a machine for measuring a behavioral difference between two groups of humans. Bots that enter the assignment pool and never convert don't just add noise — they shrink the absolute effect your test is trying to detect, inflating the sample size and calendar time needed to reach significance. This calculator quantifies that tax, generalizing the illustrative arithmetic from our bot-aware A/B testing guide. Everything is computed live in your browser; nothing here is customer data or a measured benchmark.
The dilution model
The model is deliberately simple: if a share b of sessions are non-converting bots split evenly across arms, every observed rate is the true human rate scaled by the human fraction:
Both arms scale by the same factor, so the relative lift you observe is unchanged — an 8% lift still reads as 8%. What shrinks is the absolute gap between the arms, and sample size arithmetic runs on the absolute gap. With the defaults (5.0% baseline, 8% true lift, 30% bot share), a true 5.00% → 5.40% effect is observed as 3.50% → 3.78%: the gap falls from 0.40 points to 0.28, and the sessions required per arm climb from roughly 48,000 to roughly 70,000 — about 45% more traffic, and calendar time, to find the same real effect.
The sample size formula
The calculator uses the standard two-proportion sample-size approximation — the same formula behind most published A/B test duration calculators. For an even split, sessions per arm to detect a difference between rates p₁ and p₂ at two-sided significance α and power 1 − β:
where p̄ = (p₁ + p₂) / 2, zα/2 = 1.96 for α = 0.05 two-sided, and zβ = 0.8416 for 80% power. For unequal splits the calculator uses the allocation-ratio generalization (κ = variant ÷ control share): the pooled rate becomes (p₁ + κp₂) / (1 + κ), the first radicand scales by (1 + 1/κ), and the variant arm needs κ times the control's sessions. The bot-impact number is this formula evaluated twice — at the true rates and at the diluted rates — and the difference is the traffic bots force you to spend.
Why bots inflate A/B test sample size
Dilution is the most polite way bots damage experiments, which is why this page models it: it's the best case. The full guide walks through all three failure modes — dilution (this calculator), asymmetry (sticky assignment piles repeat bot sessions into one arm, which can make a genuinely better variant read as a loser), and inflation (scrapers that trip your conversion events make whichever arm they land in look brilliant). None of this is exotic traffic: ClickStream's registry names 158 bots across 11 categories, 38 of them AI agents, and not one of them is a customer. For the broader financial argument, start with the bot-traffic business case and our companion Bot Traffic Cost Calculator.
What this calculator does not model
- Asymmetric bot arrival. Even split is the benign case. Concentrated bot traffic biases the estimate itself, not just its precision — extra traffic can't fix it; the guide shows the better arm losing.
- Converting bots. Automation that fires your conversion events inflates an arm instead of diluting both — no sample-size cure there either.
- Sequential testing and peeking. The formula is the fixed-horizon approximation; sequential designs change the constants, but bots dilute the observed gap either way.
- Your actual bot share. The 30% default is illustrative, not a measurement. Independent industry studies have consistently found automated traffic to be a substantial share of all web sessions, varying widely by industry and channel — use your own measured share.
Assumptions and defaults
- Two-sided α = 0.05 (z = 1.96) and 80% power (z = 0.8416); normal approximation to the binomial.
- Bots never convert and split evenly across arms; humans convert at the true rates.
- Per-arm sample sizes and calendar days round up.
- "Bot-free" numbers assume every session entering the test is human.
- All defaults are illustrative and user-adjustable; the URL encodes your inputs so scenarios are shareable.
Cut the tax instead of paying it
The fix isn't more traffic — it's a cleaner pool. ClickStream classifies every visitor's behavioralClass in real time, so enrollment can be gated to humans while excluded traffic still sees your canonical control page; the guide includes the complete fail-open assignment wrapper. There's a pleasant alignment, too: ClickStream bills on human pageviews, so the classification metering your bill is the one protecting your experiments — and the resulting numbers are the kind you can defend in front of a board. See pricing, or start free on Hobby: 50K human pageviews a month, no card required.