What is an A/B Test?
An A/B test (also known as a split test, split URL test, or bucket test) is a strategic effort to compare two slightly different versions of a website or marketing message against each other. The goal is to understand which of the two versions performs better, ideally leading to insights that can be more broadly applied across the organization’s marketing efforts.
For example, a website owner may want to test a new hero image on their front page. Through an A/B test, they can show two versions of the same website to two randomly selected, equal-sized segments of their target audience–one with the current hero image, the other with an alternative.
After a pre-defined test period, the results can show whether or not the shift in image has led to an improvement in key metrics like conversion rates. The website owner can then draw conclusions based on the results of that test, and continue testing other elements like a new headline, different call to action button, etc.
How to Set Up an A/B Test
The right A/B testing setup is both the most important and the most challenging piece of the puzzle. Because A/B tests can only test one page element and one variation at a time, prioritizing the correct elements to test should be the first step. Research like Google Analytics analysis, customer surveys, user testing, copy testing, competitor research, etc. can all help to understand what elements may have the biggest impact on conversion rate.
A/B testing can also be successful only when the goal of the test is clearly defined. The goal should be directly related to your desired improvement, like more clicks on a specific button or a higher check-out rate on an online store.
Another pre-testing step is running an A/A test, which pits two identical versions of the website against each other to random segments. An A/A test can ensure that your testing segment, page element, and audience section is correctly set-up, increasing confidence in A/B testing results.
When setting up the A/B test, website owners have two choices to get the variant of the original page (the “/B” in A/B test) live:
- A new URL that has already been built by the development team.
- A change on the page itself, using a tool like Google Optimize.
Making changes in the WYSIWYG editor is only possible when the A/B test concerns changes in the copy. Otherwise, hard code in the page is the best way to minimize errors in the variant and keep the results credible.
Finally, quality assurance is a vital step to ensure reliable results. Audience segments should be equal and comparable across multiple devices, browsers, etc. That might require running multiple A/B tests on the same variants to arrive at larger, more representative sample sizes.
What to Expect From A/B Testing Results
A/B testing is the most common type of performance testing in part because its premise is so straightforward. But it can only be successful when the setup is right, and when expectations are kept in check.
Most importantly, the results data can only be reliable when the test is the result of sufficient research to build a reliable hypothesis as well as clear goals. Reliable A/B tests actually only succeed about 12% to 15% of the time; higher win rates, especially when they approach and surpass 60%, tend to be a sign of weak hypotheses that don’t result in meaningful tests.
Keep in mind that the A/B test can also be a form of research in itself. Even when the test is not successful enough to be statistically relevant, it can be used as a learning opportunity to refine your hypothesis and testing goals.
Finally, it’s crucial to keep your A/B test to a single variant beyond the original version unless you have strong enough traffic to support an A/B/C test. For reference, 90% of the tests we complete for our clients are A/B tests that focus on this single variable for a higher chance of success. Contact us to start setting up your own testing mechanisms.
