When should you A/B test?
Look, the truth is I LOVE A/B tests. I want them everywhere – between the nooks of my teeth, under the couch etc. etc.
There’s a time and place for A/B tests though. We don’t need an A/B test to tell us that misspellings and poor aesthetics will damage our brand and likely lead to lower sales.
But what happens when we venture beyond the painfully obvious? Does a complete redesign of your product need an a/b test? How about a pricing change? A new feature release?
I posit that yes, these things all require A/B tests.
Because that’s how we learn the impact of things. Everything else is ego – we want our changes to be impactful so we’ll attribute any growth to our actions if we’ve failed to isolate variables.
You see, your UX changes, your price changes, your feature changes, your copy changes – none of things happen in a vacuum.
All the while you are making your changes, other people in your team are also making changes. Your Marketing team is hard at work iterating on landing pages, optimising ads, and sending out swanky email campaigns. Your product team are launching new features and improving existing features. Your UX team are reducing friction and providing a better product experience. Your CS team are learning strategies that will lead to improved customer relations.
So the only way you can know what impact your change had is to split test.
Yes, sometimes this is a drag. Many times you’ll have to hack your way to a split test using something arbitrary like a particular letter appearing in a customer’s email address (more on this in future article). Or you might have to use Google Optimise which comes with its own set of limitations and subsequent frustrations.
But if you avoid A/B tests because they’re not fun, you might as well stop brushing your teeth or washing your clothes or visiting your mother-in-law.
Without A/B tests, you’re shooting blind. This will result in 2 things:
- Crappy outcomes remaining part of your products or services.
- Future outcomes will be informed by these uninformed present actions leading to a cycle of ignorance
And the longer you do this, the more resistant you become to change. A wonderfully toxic cocktail of biases: confirmation bias, information bias, loss aversion, anchoring bias, and self-attribution bias amongst others.
So you build up your castle of sand. You probably get help from your colleagues too (we all want to be seen doing things). ‘Let’s wait and see’ sounds like procrastination.
But you fail to identify the actions responsible for most of your growth AND, just as important, you fail to spot the things holding you back.
So yeah, I think you should A/B test pretty much everything.
Then when you go to redesign your brand, you can get an actual ROI. You can say for sure whether your actions were worth it or not. This keeps you and everyone else on your team honest. And it ultimately makes your product more of what customers want and less of what you think it should be.
The day I started A/B testing everything was the day I became a truly impactful growth marketer.