What is A/B/n testing?
A/B/n testing is a more complex version of an A/B experiment that tests multiple versions of a website to random, equally sized audience segments to determine a winner. The “n” in A/B/n testing doesn’t stand for a third variable but acts as a stand-in for any number of website versions that can be tested against each other.
Beyond using more than two variations of a given website, A/B/n testing is set up like a standard A/B test. The tester determines a goal metric to test for, splits the audience into random segments, runs the test for a pre-determined period of time, and determines a winning variation based on its performance against the goal metric.
Why is A/B/n Testing Important?
A/B/n testing helps website owners understand which website design, layout, or content generates the most conversions. The ability to test multiple variations against each other simultaneously can significantly speed up the process compared to a standard A/B test, which can only test one option at a time.
In addition to determining a winner, A/B/n testing also helps you understand why pages on your website might not be performing as well as anticipated. An analysis of low-performing variations can help to draw conclusions based on low-converting page elements or design features, which can then be tested in future experiments.
Problems With A/B/n Testing
Manually selecting the number of website variations to test can lead to two potential issues in the experimentation process:
- Testing too many variations may divide traffic too much, decreasing the chances of statistically significant, relevant testing results.
- Testing too few variations may leave out the elements that actually make an impact on conversion rates, leading to missed opportunities.
A/B/n Testing vs. Multivariate Testing
Though they are similar, A/B/n testing differs from multivariate testing in that the latter automatically tests all possible combinations of page elements against each other. That process pinpoints more specifically which page elements might lead to better or worse conversion rates.
The ability of multivariate testing to test every possible combination of page elements makes it helpful in pinpointing exact changes. But it also significantly increases the complexity of its setup and a high amount of traffic for statistically relevant results. The results of A/B/n testing aren’t as comprehensive, but easier to set up and allow for a manual choice in which options to test.
When to Use A/B/n Testing Instead of Multivariate Testing
Because of their simpler setup, A/B/n tests work best in situations where making a quick decision about a website matters most. The test helps to determine the best overall page out of a few defined alternatives when they’ve already been built in the development stage.
Multivariate testing is the better option when looking for not an overall ‘winner’ page, but determining exactly which elements on a given page most contribute to traffic, conversions, or other goal metrics. They take longer, and the results lead to more granular decisions, but can lead to broader conclusions about the effectiveness of individual site elements like buttons, headlines, or layout choices.
How to Set Up an A/B/n Test
Setting up a successful A/B/n test requires a few carefully thought-out steps:
- Determine what page you want to test in an A/B/n scenario. The best choice tends to be a popular page with high traffic and conversion numbers like your homepage, eCommerce product page, SaaS pricing page, etc.
- Define the number of page variations to test based on the average traffic on the page you’ve chosen. Ideally, each variations should be expected to receive at least 1,000 visits during your testing period.
- Define your goal metric. Most often, that goal metric will be focused on conversion rate. But it could also include clicks on the ‘Add to Cart’ button, new live chats opened, and other key actions.
- Design the page variations based on different layout types and styles in consideration. Each variation should be different enough to determine a clear winner after the test period is over.
What to Expect From A/B/n Test Results
Once the test is complete, a successful experiment will show a clear winner and loser with a high confidence interval. Avoid drawing conclusions that are too broad from this experiment; the winner will likely outperform the other variations on a permanent basis, but it might not mean that other pages built similarly will perform just as well.
Successful A/B/n tests are a great way to quickly pit different page variations against each other before moving forward with a final page design. They can be complex to set up, but you’re not on your own. Contact us for help on strategic conversion rate experimentation and optimization.
