What Is A/B Testing and How Do You Run a Test You Can Trust?

An A/B test shows two versions of a page, ad or email during the same period to visitors split randomly into two groups, and measures which version performs better on a single goal chosen in advance. How to run one, in short: write a hypothesis, choose one primary metric, calculate the number of visitors you need up front, run the test untouched until it reaches that number, and read the result by looking at the confidence interval, not just the percentage difference.

Skip any step in this sequence and the test will still show a "winner", but that winner is often just chance. Below we go through each step and the most common mistakes.

What is A/B testing for?

A/B testing replaces guesswork with evidence. Ideas such as "if we make the button green, more people will click" or "if we put the price in the headline, more people will fill in the form" are tested against real visitor behavior. Because both groups arrive during the same period and from the same traffic sources, the difference between them can largely be attributed to the change you made.

Three related concepts are often confused:

• A/B/n test: More than two versions are tried at once. Because each version splits the traffic, the total number of visitors needed goes up.
• Multivariate test: Combinations of several elements, such as headline, image and button, are tested together. It needs very high traffic.
• Split URL test: The versions live at different addresses; it is used to compare a completely redesigned page against the existing one.

How do you run an A/B test?

1. Find the problem with data. Identify the step in your analytics funnel with the biggest drop-off, and the friction point using heatmaps and session recordings. We explain how to read heatmaps in our heatmap analysis guide.
2. Write a hypothesis. "If we show the shipping cost in the first step of checkout, checkout completion will increase because worry about surprise costs will go down." A good hypothesis says what you will change, why, and on which metric.
3. Choose one primary metric. Such as purchases, form submissions or checkout completion. Track metrics such as clicks and add-to-cart only as supporting metrics.
4. Calculate the sample size you need. Enter your current conversion rate and the smallest difference you want to detect into a sample size calculator. The number it gives also determines how long the test will run.
5. Split traffic randomly and evenly. The same visitor needs to see the same version every time they return; the testing tool handles this with a cookie or user ID.
6. Fix the duration in advance. Do not stop the test until the calculated sample is reached and at least one or two full weeks have passed, and do not make other changes to the page in the middle.
7. Document the result. Whether it wins or loses, write the hypothesis, the result and what you learned in a test log; your next hypothesis comes from there.

How much traffic and time does an A/B test need?

The number of visitors you need depends on four things: your current conversion rate, the smallest difference you want to detect (minimum detectable effect), statistical power and the significance level. Evan Miller's free sample size calculator asks for exactly these four inputs.

The rule is simple: if your conversion rate is low and the difference you are looking for is small, you need a lot of visitors. On a page with a 3 percent conversion rate, reliably seeing a 20 percent relative lift requires, by the standard calculation, more than 10,000 visitors per version. On a site with a few thousand visits a month, that test will take months. In such cases, it makes more sense to test bold hypotheses that change the offer, the way price is presented or the page structure, instead of small color changes; large differences show up with less data.

As for duration, weekday and weekend behavior differ, so run the test in full weeks. Campaign periods, public holidays and big sale days distort the result; test outside these periods where possible.

How do you read the result correctly?

The most common mistake is checking the results every day and stopping the test as soon as significance appears. According to the calculation in Evan Miller's article "How Not To Run an A/B Test", a test that is monitored continuously and stopped as soon as it looks significant can push a false positive rate that is 5 percent on paper up to as much as 26.1 percent. The fix is to set the sample size in advance and stick to it.

When reading the result, watch for these points:

• A percentage difference alone is not enough. If the confidence interval includes zero, you cannot say "B is better".
• Do not go hunting for segments afterwards. When the overall result is not significant, sub-segments such as "it won among female mobile users" are usually chance. If you defined a segment before the test, evaluate it separately.
• Account for the novelty effect. A new design can attract attention in the first days simply because it is different, so look at the whole period, not the early days.
• A null result is still a result. Learning that a change made no difference keeps you from investing in the wrong direction.

Which tools can you use for A/B testing?

Google Optimize and Optimize 360 were shut down on 30 September 2023. Google said it was turning to third-party tools that integrate with Google Analytics and listed AB Tasty, Optimizely and VWO as partners. Today, website tests are run with tools of this kind or with server-side testing solutions built into your site's infrastructure.

On the advertising side, platforms have their own experiment tools. In Google Ads, custom experiments let you assign part of a campaign's budget to a test version; Google recommends a 50 percent split for the best comparison, and although five experiments can be scheduled for a campaign, only one can run at a time. For subject line and send time tests in email, see our post on email A/B testing.

Does A/B testing hurt SEO?

Not when it is set up correctly. Google Search Central's guidance on A/B testing sets four rules: do not show Googlebot different content from what visitors see (cloaking), use rel="canonical" pointing to the original page on test versions that live at a different URL, use a temporary 302 redirect rather than a permanent 301 if you need to redirect, and do not run the test longer than necessary. Google states that it may interpret an experiment that runs unnecessarily long as an attempt to deceive the search engine. When the test ends, make the winning version permanent and remove the test code.

The short version

• An A/B test compares two versions with randomly split traffic during the same period.
• Do not start without a hypothesis and a single primary metric.
• Calculate the sample size up front; do not stop early because of what the results look like.
• With low traffic, test hypotheses that could create large differences, not small tweaks.
• Use canonical tags and 302 redirects on test pages, and do not keep the test running longer than needed.

If you would like to set up the test plan and measurement infrastructure together, take a look at our conversion optimization service.