Not every feature is A/B test worthy

A/B testing is a great tool and I myself am a strong A/B test believer. But as with any tool, it is not a one-size-fits-all instrument, hence it has to be used carefully and not to be relied on in 100% of cases.

For instance, it doesn't make much sense to test small UX improvements that don't directly affect important product metrics. And it's hard to test big and complex product shifts that reflect changes in the product vision.

What is suitable for testing is a separate feature, that:

• Is big enough to cause a decent shift in an important metric;
• Is small enough so you can easily locate it;
• Is close to an important conversion point, so you can be sure it affects it;
• Is on a critical user path so it is worth wasting your team's time on it;
• Doesn't require a huge additional development for an A/B test itself.

Payment or signup forms are perfect candidates for A/B testing. Improving all the communications across users' journey - not that much.

In Job Today we are constantly running monetisation and acquisition tests on various audiences and it allows us optimise paid plans, increase revenue and improve the top of the funnel. Other features are only A/B tested if we decide it's necessary.

If something is broken, outdated, or causes users mistakes or confusion - just fix it, no A/B test is needed.

If basic functionality is missing or existing one goes against best practices without a bold reason (e.g. a welcome email is missing, there's no link to the mobile app from your website, a user can lose their content in one misclick), just implement it without split testing.

If you have legacy features that are not aligned with your product vision, just kill them, no need to test (just make sure your vision fits user needs ;)).

If you want to make UX improvements in the product functionality that you believe will help users (say, you saw it during user research), just go for it, no need to run A/B testing.

When speaking about A/B testing, people usually forget to mention its cost, but it's a pretty expensive endeavor.

If your company is not the size of Airbnb yet, you likely don't have an automated A/B test system that everybody in the company can safely and easily run dozens of tests simultaneously. You likely need engineers to run a test, configure audiences and include the right features, and then somebody (QA or even yourself) to test it all. In many cases, your team needs to implement additional functionality to support an A/B test for a certain feature and then remove it after the test is over.

It's easy to mess up in the process and end up rerunning the test. And even if you don't mess up, the process is still time-consuming. You waste developers' and QA's time and you delay the moment of getting the results you need to learn and iterate. The more features you want to test, the longer the queue you have, and the more of a bottleneck it becomes, which inevitably causes more delays in development and learning.

Another trap is that you can over-rely on them, get reluctant to take a step without A/B, and start testing random stuff without a strategy and direction. It's easy to get stuck in meaningless optimisations around a local extremum instead of moving the product forward relying on your intuition and product sense.

It feels safe to be able to test everything. So you stop making bold decisions and find creative ways to validate ideas cheaper.

And don't forget about situations where a split test doesn't tell you which version performs better (which happens more often than we tend to admit). We end up with no learning, wasted time, and anxiety because we haven't decided which version we want to keep as we relied on A/B testing results.

There're lots of ways to validate that we are doing the right things. The A/B testing is only one of them and not the cheapest one.

User testing before the implementation, data analysis, and user interviews after the launch, monitoring customer support tickets, NPS, and reviews - are way cheaper and faster ways to do it.

You can even do slow release roll-outs and compare user behavior between cohorts that will give almost the same visibility as A/B testing without spending developers' time.

What is definitely good about A/B tests - it's really hard to forget about a running test, so it forces you to explicitly check the results and chose a winning version. When you just implement a feature, it's very easy to forget about it after it moves to the "Done" status. So this situation should be managed. Either assign tickets to yourself or manage a separate board together with the Analytics team if you have one.

I'm testing a board in Notion right now, and when I don't forget to update it, it works fine :) The idea is to keep a feature in the "In testing" column after it's released until you validate that it works as expected (or not) and decide what to do: keep, kill or iterate.

A/B testing is not a one-size-fits-all instrument, nothing is. There are many situations when it really helps to learn and optimise conversions, monetisation options, and critical parts of a user flow. But we shouldn't forget that it comes at a cost and in many situations, A/B tests just slow us down and create a false sense of control, limit our capacity to validate ideas quickly and cheaply, and build conviction around our ideas and use our brain.