Logo

MonoCalc

/

Ad Test Sample Size Calculator

Social Media
The rate your control already gets, from your own account.
A percentage of the baseline: 25 means a rate 25% higher than the control's. No default: the smallest change worth acting on is your business judgement, not something this page can know.
The same number means different things in each mode.

—

Derived from your baseline and effect — check it reads as you intended.
Choose it before the test starts, not after.
Detects a move in either direction. Critical value Φ⁻¹(1 − α/2).
Entered as a percentage. Must be above α and below 100%.
Only used for the duration estimate. Leave blank and no duration is shown.
Enter the baseline conversion rate your control already gets.
Enter a minimum detectable effect — the smallest change you would act on.

About This Tool

Ad Test Sample Size Calculator – Plan Before You Spend

Two creatives go live on Monday. By Wednesday one is ahead, by Friday the other is, and at some point somebody declares a winner because the dashboard looked right that morning. The test never had a chance of answering anything, because nobody worked out how much traffic it would take. An ad test sample size calculator settles that question in advance: given the rate you already get and the smallest improvement you would actually act on, how many impressions does each variant need before the comparison means something?

The formula, and what each piece is doing

This is the normal-approximation formula for comparing two independent proportions. Everything in it is a number you chose.

p̄ = (p₁ + p₂) / 2 0.045 ( z_α·√(2p̄(1−p̄)) + z_β·√(p₁(1−p₁)+p₂(1−p₂)) )² n_arm = ─────────────────────────────────────────────── (p₂ − p₁)² z_α = Φ⁻¹(1 − α/2) two-sided 1.959964 z_β = Φ⁻¹(power) 0.841621 n_arm at 4% → 5%, α=0.05, 80% power 6,744.93 → 6,745

Two details in there cause most of the confusion. The first is that z_β is the quantile of the power, not of β — feeding it 1 − power flips the sign and quietly halves the answer. The second is that n_arm sizes one group. A test has two, so the traffic it really consumes is double: 6,745 per arm is 13,490 impressions in total. The calculator above prints both figures next to each other because showing only the first is the standard way this class of tool misleads people.

The minimum detectable effect is a business decision

The minimum detectable effect is the smallest change worth the creative work and the budget to chase. Nothing in a calculator can know that number — it depends on your margins, your production costs and what you would actually do differently if the variant won. That is why the field ships empty here and why no value is offered as a starting point. It is also the most consequential input on the page, and the one people set carelessly.

The cost of setting it small is not linear, it is quadratic: n scales with 1/(p₂ − p₁)². At a 4% baseline with α = 0.05 two-sided and 80% power, detecting a +25% relative lift takes 6,745 per arm, while detecting a +10% relative lift takes 39,475 per arm. An effect 2.5 times smaller costs 5.9 times the traffic. The curve on this page draws exactly that wall.

Fix the stopping rule before the test starts

Peeking breaks the guarantee
A significance level assumes one look at the data. Checking every morning and stopping the moment the result crosses the line gives the noise many chances to cross it, so the real false-positive rate climbs far above the α you picked. Computing the sample in advance is what fixes the stopping point: run to the number, then analyse once.

This is the whole reason the planning step exists. A test with no pre-declared end is not a test, it is a search for the morning on which the numbers look good. Once the planned sample is reached, the analysis is a separate job — the companion Ad A/B Test Significance Calculator takes the observed counts and produces the p-value and the confidence intervals. This page deliberately refuses to accept conversion counts and never prints a verdict.

“No significant difference” is not “no difference”

An inconclusive test says your data could not separate the two options at the precision you asked for. It does not say the creatives perform identically. An underpowered test will usually come back inconclusive whether or not a real effect exists, and on the occasions it does show a winner, the size of the win will be badly overstated — only the luckiest noise clears the bar, so the estimate that clears it is inflated. That is the practical danger of running a test that was never big enough.

When the required sample is more traffic than you have

Sometimes the honest answer is that the test cannot be run. That is worth knowing before the budget is committed rather than after. The levers are real but limited: accept a larger detectable effect, since only a bigger difference is findable at your volume; lower the power and accept missing real effects more often; or test a change with a bigger expected impact than a headline tweak. Running the underpowered version anyway is the one option that spends the money and returns nothing.

What the number is, and is not

The figure is an approximation. The normal approximation to a binomial comparison holds well at the rates and volumes paid-social tests run at, but it degrades when the expected number of conversions per arm is small — below roughly ten expected successes and ten expected failures in each cell, the tool flags it and the true requirement is likely somewhat larger. It is also specific to two arms and a binary outcome. Testing three creatives at once, or measuring revenue per visitor rather than a conversion rate, needs a different formula altogether.

Frequently Asked Questions

Is the Ad Test Sample Size Calculator free?

Yes, Ad Test Sample Size Calculator is totally free :)

Can I use the Ad Test Sample Size Calculator offline?

Yes, you can install the webapp as PWA.

Is it safe to use Ad Test Sample Size Calculator?

Yes, any data related to Ad Test Sample Size Calculator only stored in your browser (if storage required). You can simply clear browser cache to clear all the stored data. We do not store any data on server.

How does this ad test sample size calculator work?

You give it three things: the conversion rate your control already gets, the smallest change you would actually act on, and how confident you want to be. It derives the target rate from your baseline and your minimum detectable effect, then applies the standard normal-approximation formula for comparing two independent proportions — n = (z_α·√(2p̄(1−p̄)) + z_β·√(p₁(1−p₁) + p₂(1−p₂)))² / (p₂ − p₁)². The critical values come from the inverse normal CDF, so any α and any power work rather than just the two or three a lookup table would allow. Everything runs in your browser on the numbers you type; no ad account is connected and nothing is sent anywhere.

Why is the answer per arm, and why must I double it?

The formula sizes one group. A test has two — the control and the variant — and both need to reach the number before the comparison is powered, so the traffic the test actually consumes is twice the per-arm figure. Reporting only the per-arm number is the single most common way sample size tools mislead people: a plan that looks like 6,745 impressions is really 13,490. The calculator shows both figures side by side for exactly this reason. If you are testing three creatives rather than two, this formula does not cover it — multiple comparisons need a correction it does not apply.

What is statistical power, and is 80% a rule?

Power is the probability that your test detects a real difference of the size you specified, assuming that difference genuinely exists. At 80% power, a real effect gets missed one time in five — the test comes back inconclusive even though the variant truly is better. 80% is a convention, not a rule and not a requirement: it is simply where the discipline settled as a trade-off between certainty and traffic. Raising it is legitimate and costs traffic, which is what the power sensitivity strip on this page is for. The number is your call, and it should be made before the test starts rather than after.

Why does detecting a smaller effect cost so much more traffic?

Because the sample size scales with the inverse square of the difference: n is roughly proportional to 1/(p₂ − p₁)². Halving the effect you want to detect therefore roughly quadruples the traffic required. At a 4% baseline with α = 0.05 two-sided and 80% power, detecting a +25% relative lift needs 6,745 per arm, while detecting a +10% relative lift needs 39,475 per arm — an effect 2.5 times smaller costs 5.9 times the traffic. This is why the minimum detectable effect is the most consequential number on the page, and why the curve is worth reading before you commit to a plan.

Why does peeking at the result early break the test?

The significance level you choose assumes exactly one look at the data. Checking every morning and stopping as soon as the result crosses the threshold gives the noise many chances to cross it, so the real false-positive rate climbs far above the α you selected — a test peeked at daily can reach “95% significant” by chance far more often than 5% of the time. Computing the sample size in advance is what fixes the stopping rule: you run to the planned number, then analyse once. Stopping early because the result looks good is not an efficiency, it is a way of manufacturing a winner.

What if the required sample is more traffic than I will ever get?

That is a real answer, not a failure — it tells you the test cannot settle the question at the precision you asked for, and you have learned that before spending the budget rather than after. Your options are to accept a larger minimum detectable effect, since only a bigger difference is findable at your volume, to lower the power and accept missing real effects more often, or to test something with a bigger expected impact than a creative tweak. What does not work is running the underpowered test anyway: it will usually come back inconclusive, and when it does show a winner the estimate of the win will be badly inflated.