Let’s test a button color! I hear this a lot with clients. While that can sometimes be a valid beginner AB test, your web designer and UX specialist likely chose the current color based on years of sound research. That likely makes it an unfavorable candidate for a good test.
So, what makes a good test then? What’s the secret sauce?
The Attributes of a Good AB Test
It Tests a Component/Page that Influences Conversion
A conversion is quantifiable and definitive, providing concrete data on whether the change positively or negatively impacts user behavior and outcomes – and potentially revenue.
It is User-Centric
Make sure your tests deliver value to the user’s experience. Create a test based on site analytics, heatmaps, user testing, and other tools to ensure your test aims to improve a user painpoint, and they will reward you with increased engagement and leads – or a lesson in what they do not want. Even a failed test has value.
It Has an Adequate Sample Size
Your test needs enough visitor traffic to hit statistical significance. Check your analytics and ensure you are testing a high-traffic page or collaborating with marketing to find a page that will drive campaign traffic. Setting up your AB tests with a confidence level of 90-95% is recommended. Without adequate traffic, your test will take significant time to hit the above 90% milestone – or it may not at all. The larger your sample size, the more you can trust the results. Many testing tools will calculate this for you and provide an estimate for how many hours or days the test will take to complete. If you want to manually determine what sample size you need for a particular test, there are many calculators online, including Sample Size Calculator (Evan’s Awesome A/B Tools) (evanmiller.org).
It is Easily Configurable
This is for those who want to keep things as easy as possible. Consider whether a content author can test the element requested by the business. If a developer is required to configure the test, it may take additional time to get through the development workflow. If the test is difficult to set up, it may be a more efficient use of time to pivot to a more straightforward test that may yield results with similar takeaways. It’s more about what you are trying to learn about your visitors.
It is Run at the Right Time
Choosing your test date could be just as important as the others. Running a test during a time with a seasonal increase in traffic could create false confidence that can’t be replicated in other months.
Follow these suggestions, and you’ll be on the right track towards better tests and a more effective website!
