Most email A/B tests are just coin flips with extra steps
Here's a common email "experiment": split 400 subscribers in half, send subject line A to one side and B to the other, wait two hours, crown the one with more opens, and add the insight — "questions beat statements!" — to the team playbook forever.
That's not a test. At those numbers, it's a coin flip with a dashboard. A 24%-vs-21% open split on 200 people per arm is well inside random noise; rerun it tomorrow and it flips. Half of email's folk wisdom is built on winners like these.
You don't need a statistics degree to test honestly. You need four habits.
1. Respect the sample size
Small differences need big samples to be real. As rough intuition: detecting a modest lift in open rate takes at least a thousand or so recipients per arm — and detecting differences in clicks takes several times more, because click counts are so much smaller. (Free significance calculators are everywhere; two minutes with one before declaring a winner is the entire discipline.)
Consequences of taking this seriously:
- Small list? Test only things that could produce large effects — a completely different offer or angle, not "Hi" vs "Hello." Tiny lists can detect earthquakes, not tremors.
- A result that isn't significant isn't a "small win." It's a shrug. Record it as one.
2. One variable, or you learned nothing
If version A has a different subject line, a new hero image, and a reworded CTA, then whichever wins, you don't know why — you've tested "email A vs email B," an answer that expires the moment either email changes. Change exactly one thing per test. Slower, and the only version that produces reusable knowledge.
3. Measure the metric you actually care about
Two traps here. First, opens are machine-inflated (Apple's prefetching — see our MPP post), so subject-line tests read through fog; opens remain usable comparatively between arms, but prefer clicks whenever your volume allows. Second, and sneakier: the winner by opens can be the loser by results. Clickbait subject lines reliably win opens and lose downstream — the curiosity-gap subject earns the open, disappoints, and depresses clicks and future engagement. Judge tests by the deepest metric your sample size can support: clicks over opens, conversions over clicks.
And give tests time. A two-hour verdict measures your fastest openers, who aren't a random sample of your list. Let a test run 24 hours minimum before reading it.
4. Test where the leverage is
Ranked by how much a win is worth:
- The offer and angle — what the email proposes and how it's framed. 10x the impact of any wording tweak, and the thing least often tested.
- The welcome email — every improvement compounds across all future subscribers. The single best home for your testing effort.
- Send-day/time and frequency — real, findable effects at list level (per-person "optimal times" mostly died with reliable open data).
- Subject lines — fine, fun, and the most over-tested element in email relative to its payoff.
One more compounding rule: keep a log. Hypothesis, variant, sample size, result, significant or not. Without it, teams re-run the same tests annually and remember only the flukes. The log is the asset; individual tests are just entries.
The bottom line
A/B testing works — it's the theater of A/B testing that wastes everyone's time. Sample sizes that can support a conclusion, one variable, real metrics, patience, and a written log. Do less testing, honestly, on the things that matter, and your playbook fills with knowledge instead of coin-flip folklore.