Bot-Aware A/B Testing: Experiments That Converge on Humans

@clickstreamhq/signals is a developer preview: it's published to npm on the 0.1.0-alpha line under the alpha dist-tag. Every API in this article is the real, current surface — verbatim-checkable against the package. And every number in the dilution examples below is illustrative arithmetic, chosen to be easy to verify with a calculator. None of it is customer data.

Why Bot Traffic Breaks A/B Testing

An A/B test is a machine for measuring a behavioral difference between two groups of humans. Every non-human session that enters the assignment pool degrades that machine in one of three ways:

• Dilution. Bots that never convert shrink the absolute gap between your arms, so the same real effect takes far more traffic — and far more calendar time — to reach significance.
• Asymmetry. Bots don't arrive uniformly. A monitoring service pings one URL on a schedule; a scraper fleet hammers your catalog; deterministic assignment (which is correct for humans) makes every one of those repeat sessions pile into whichever arm its cookie was first bucketed into.
• Inflation. Some automation fires your conversion events. A scraper that walks every product page trips a "viewed 3+ products" micro-conversion at close to a 100% rate. Whichever arm it lands in looks brilliant.

This isn't exotic traffic. ClickStream's registry names 158 bots across 11 categories — search crawlers, SEO tools, social preview fetchers, uptime monitors, scrapers, scanners, automation frameworks, stealth bots, kiosks — and 38 of them are AI agents, a lane that's growing as answer engines browse on users' behalf. Not one of them is a customer. All of them are sessions, as far as a naive experimentation setup is concerned.

An 8% Lift, Diluted: the Illustrative Math

Suppose your checkout variant genuinely converts humans 8% better: control converts at 5.0%, the variant at 5.4%. To detect that 0.40-point gap at the usual bar (two-sided α = 0.05, 80% power), the standard two-proportion formula asks for roughly 48,000 sessions per arm.

Now let 30% of assigned sessions be non-human traffic that never converts, split evenly across both arms — the benign case. Your measured rates drop to 3.50% and 3.78%. The relative lift still reads 8%, but the absolute gap you're trying to estimate shrank from 0.40 points to 0.28, and the same formula now asks for roughly 70,000 sessions per arm — about 45% more traffic to find the same real effect. A two-week test becomes a three-week test, and a modest true lift that would have cleared significance now reads "inconclusive."

The asymmetric case is worse, because no amount of extra traffic fixes it. Same setup — 10,000 human sessions per arm, 500 conversions in control, 540 in the variant — but this time a handful of high-frequency monitors and scrapers, sticky-assigned by their cookies, contribute 1,000 zero-converting sessions all to the variant. The variant now measures 540 / 11,000 = 4.91%, against control's 5.00%. Your genuinely better arm reads as a loser. Significance math assumes the two populations are exchangeable; concentrated bot traffic quietly violates that assumption, and the p-value will not warn you.

The Gate: behavioralClass === 'human'

ClickStream classifies every visitor on two independent axes, both exposed to page code through the Signals visitor intelligence API:

• visitor.bot — network-level classification: user-agent and infrastructure matching against the registry, surfaced as { isBot, category, name, score } (category is one of the 11 lanes, e.g. search_crawler or stealth_bot; name is e.g. "Googlebot").
• visitor.behavioralClass — a behavioral verdict built from the session so far: 'human' | 'suspicious' | 'likely_bot' | 'bot'. It's independent of the network axis on purpose — automation running a clean user agent can be isBot: false yet score likely_bot behaviorally. The two-layer botScore/humanConfidence engine behind it is documented in the Bot Evasion Atlas.

For experiment assignment, gate on behavioralClass === 'human' and exclude everything else — suspicious, likely_bot, and bot alike. The reasoning is a cost asymmetry: admitting one bot pollutes the statistics for everyone, while excluding a misclassified human costs exactly one person the variant experience — they see control, which is your canonical page. When failure modes are that lopsided, you tune the enrolled pool for precision, not recall. (If you want the strictest possible pool, also require !visitor.bot.isBot, which additionally drops UA-flagged sessions whose behavior reads human — such as real people behind flagged corporate proxies.)

And because no classifier is perfect, ClickStream gives operators a correction lane: the mark-it-human override reclassifies a wrongly flagged visitor from the dashboard.

The Assignment Wrapper: Fail Open to Control

As always with Signals, configure({ apiKey }) comes before any read. The browser key is public by design and domain-gated, so it's safe to ship in page code:

import { configure } from '@clickstreamhq/signals';

configure({
  apiKey: 'cs_live_xxx',             // browser key: public by design, domain-gated
  endpoint: 'https://t.example.com', // your verified first-party tracking domain
});

The wrapper itself is built around getVisitorOrNull() — the fail-open read — and it never throws. Every path that can't produce a live, human-classified snapshot resolves to control, unenrolled, with a reason string for the audit log:

import { getVisitorOrNull } from '@clickstreamhq/signals';

const CONTROL = 'control';

// Deterministic bucketing — FNV-1a over experiment + visitor, never
// Math.random(), so a returning visitor keeps the same arm.
function bucketOf(key, buckets) {
  let h = 0x811c9dc5;
  for (let i = 0; i < key.length; i++) {
    h ^= key.charCodeAt(i);
    h = Math.imul(h, 0x01000193) >>> 0;
  }
  return h % buckets;
}

export async function assignVariant(experimentId, variants) {
  const visitor = await getVisitorOrNull();

  // 1. Fail open: Signals unavailable, blocked, or rate-limited
  //    past the stale-reuse window -> control, not enrolled.
  if (!visitor) {
    return { variant: CONTROL, enrolled: false, reason: 'signals_unavailable' };
  }

  // 2. Placeholder snapshots are not evidence. While a visitor is
  //    initializing or Signals Coverage is exhausted, the server returns
  //    a conservative placeholder: all scores 0, stale: true, and
  //    behavioralClass 'suspicious'. Check staleness BEFORE the class
  //    check, or a coverage outage would silently exclude everyone.
  if (visitor.stale || visitor.pending) {
    return { variant: CONTROL, enrolled: false, reason: 'snapshot_not_live' };
  }

  // 3. The gate: humans only. 'suspicious', 'likely_bot', and 'bot'
  //    all resolve to the control experience.
  if (visitor.behavioralClass !== 'human') {
    return {
      variant: CONTROL,
      enrolled: false,
      reason: 'excluded:' + visitor.behavioralClass,
    };
  }

  const bucket = bucketOf(experimentId + ':' + visitor.identity.visitorId, variants.length);
  return { variant: variants[bucket], enrolled: true, reason: 'human' };
}

Step 2 is the detail teams miss. The degraded placeholder context (served with coverageMode: 'degraded' when the Signals Coverage budget is exhausted, or while a brand-new visitor's first scored event is still in flight) deliberately carries behavioralClass: 'suspicious' so that score-gated personalization fails closed. In an experiment gate, that conservatism would masquerade as a bot surge — so the wrapper checks stale and pending first and files those sessions under a separate reason. The broader defensive catalog lives in Fail-Open Personalization Patterns.

Usage follows one rule: only enrolled sessions emit an exposure event. Your analysis population is exactly the set of exposures, so excluded traffic never touches the denominator:

const { variant, enrolled, reason } = await assignVariant(
  'checkout_cta_v2',
  ['control', 'treatment_sticky_cta'],
);

renderVariant(variant); // excluded traffic renders control — a complete page

if (enrolled) {
  logExposure({ experimentId: 'checkout_cta_v2', variant });   // your experiment system
} else {
  countExclusion({ experimentId: 'checkout_cta_v2', reason }); // your audit counter
}

One latency note: a Signals snapshot is a single REST round trip, documented at roughly 50–150 ms — fine for post-paint assignment, which is where most client-side experiments already run. The fail-open path bounds the worst case: if the read doesn't come back, the visitor simply sees control.

Log Exclusions for Audit

A gate you can't observe is a gate you can't trust. The reason strings exist so that every assignment lands in one of three auditable buckets — enrolled, excluded:*, or unavailable — and the ratios become a monitoring surface:

• Reconciliation. Assigned = enrolled + excluded + unavailable. When someone asks why the dashboard shows more sessions than the experiment analyzed, the answer is a query, not a shrug.
• Drift detection. If excluded:suspicious doubles week over week, something changed — a scraper fleet found you, a traffic source degraded, or classification shifted. You want to see that before it shows up as a mysteriously flat test.
• Coverage visibility. A spike in snapshot_not_live tells you reads are degrading to placeholders — a capacity conversation, not a traffic-quality one.

There's a pleasant alignment here: ClickStream's billing counts human pageviews, so the classification that meters your bill is the same one gating your experiment. The pool your test converges on is, to a first approximation, the pool you're actually paying to measure.

Excluded Traffic Still Gets Control

The wrapper never returns "nothing." Excluded and unavailable sessions get the full control experience, and that's a design position, not a shrug:

• Search crawlers should see the canonical page. Serving experimental variants to Googlebot invites indexing your test copy; serving it something broken risks being read as cloaking. Control is your canonical experience.
• AI agents are reading on a person's behalf. The 38 named AI agents in the registry feed answer engines that describe your product to real buyers. A degraded page for ClaudeBot or PerplexityBot becomes a degraded answer about you.
• Misclassified humans lose nothing important. The worst case for a false positive is the default page — complete, functional, canonical.

One guardrail to keep explicit: this gate decides who enters the measurement, never what anyone pays. Using behavioral scores to show different visitors different prices is a dark pattern ClickStream's own guidance rules out — don't score-gate pricing, in experiments or anywhere else.

The Bottom Line

• Bots in the pool cost you either traffic or truth — evenly split, an illustrative 30% bot share turns a detectable 8% lift into a ~45% longer test; concentrated in one arm, it can flip the verdict outright.
• Gate enrollment on visitor.behavioralClass === 'human' — the behavioral axis catches clean-UA automation that network checks miss.
• Check stale/pending before the class check — placeholder snapshots are conservative by design, not evidence of bots.
• Fail open to control, log every exclusion — getVisitorOrNull() plus reason strings make the gate observable and the failure mode boring.

An experiment that can't say who's human can't say what won. Gate the pool, audit the gate, and let everything else see control.