How I became an A/B-testing addict

Every product manager says "You must use A/B testing!". But do you know somebody who really does it on a day-to-day basis? I'm happy for you if you do :) I didn't realise the true value of A/B testing until I started working at Nautal – marketplace with a volatile conversion rate that changes a lot depending on the season, region and even weather.

Any business, related to sport, tourism or any outdoor activities can feel this instability. For example, in our nautical sector the highest conversion is in summer. In September it starts to go down. An average check behaves the opposite way as well as a decision-making time. In winter it takes days or even weeks for people to decide on the deal but in summer they need a boat right away. The percentage of mobile and desktop users also varies depending on the season.

And the weather – oh yes, the weather is a pain.

Changes are fast and dramatic. One week can naturally perform better or worse up to 20-30% than another one. In these circumstances, it's practically impossible to say whether your UX improvement made any difference. So without A/B testing, you are almost blind.

Previously I had heard about Optimizely – popular tool for A/B testing with an awesome knowledge base. But it is so expensive that there aren't even any prices on the website :) it is thousands of dollars per year.

So we chose Google Optimize instead. It has more or less similar functionality as Optimizely, but it is free. The cool thing is that it works together with Google Analytics where you can view detailed reports.

Ok then, let's start from the simple things and see how it works.

Have you heard the phrase «A button can be any color, if it is red»? So now we can prove it right or wrong.

Optimize asks what page you want to use as a base for configuring an experiment and uploads it to the visual editor. There you can choose any block and specify its color, size, paddings, border.... in short, everything that can be done using CSS.

You can change headers, captions, links… basically all the static content on the page.

Together with the targeting of the audience, this feature can be used to customize the page for the user. For instance, you can show personal headlines to people who have come from advertising or visitors from a particular region, repeat visitors or users of mobile devices.

Important! All of this must be checked and tested carefully because it is very easy to break something, e.g. the website localisation. Also, it is important to be careful when changing the dynamic content and interactive elements. Say if you change a search field placeholder you can damage all the search.

The general rule is this – at first, the original page uploads with all the static and dynamic content. Then all Optimize changes apply. We just have to take it into consideration.

One of my favourite types of experiments – block sorting. E.g. the position of blocks on the landing page or product page can influence the conversion rate pretty strongly.

Drag and drop blocks wherever you want. You can put a block in any place of the page (it usually means you damage an HTML structure) or use a default option to reorder the blocks of the same type – list items or one level divs. At this point, you will imbue with respect (or not) to your frontend developer and see how well the layout is done.

Reordering of blocks can give a very good increase in conversion rate. Plus, this experiment will tell you whether or not this block is important for users.

Also, you can reorder blocks in one experiment variant and completely remove one of the blocks in another and see what will happen.

For example, on our yacht description page, there are two blocks - "Secure booking" and "Contact the owner". Nobody knew whether they are necessary or not and which of them is more important.

That's why I run an A/B test with three variants – original, with reordered blocks and without the security block.

A variant with reordered blocks constantly shows the best conversion rate. An interesting thing is that a variant without a security block performs worse than the original variant. So we can't delete it but it can be a good idea to switch the blocks.

Say, you add something to the page and want to test whether it improves conversion rate or not. It is super easy to test this in Optimize. Just remove a new block from one of the variants and launch an experiment. In a few days compare the results and start pulling out your hair celebrate.

Sometimes you don't really see a big difference between variants and the results are contradictory. It means that either everything is fine with an interface block and even your improvement can't damage it :) or nobody is using the block in the first place.

You can try to delete the block or to move it to a different place and see how it will influence the conversion rate.

But sometimes you see very clearly that one variant performs better than the other. That means you are on the right track and it's a good idea to implement your improvement on all the website.

In order to get a statistically significant result, it's better to wait for a couple hundreds of conversions. Plus, Optimize itself recommends running experiments for at least two weeks in order to exclude an impact of intra-week fluctuations.

Say, an A/B experiment confirmed that our idea was good and a new variant performs better than an old one. What should we do? We can (and in many cases need to) formulate the task for developers.

Even if we want a new feature in production right away, we know how long it usually takes to develop, test and deploy it. So in order not to waste time – we can just switch the best variant in Optimizer to 100% of the audience. This is the way to deliver the best design to production fast and at the same time give your IT team time to develop and test everything properly.

Google Optimize has a pretty advanced constructor for targeting the A/B experiment audience – 11 groups of rules for making magic.

I have been using two categories so far:

I found it not quite informative how Optimize shows a list of running and finished experiments. First – it doesn't show experiments of all the domains in one list and I have different language versions of the site working on different domains. Second – a list doesn't show which variant performs better and what conclusions can be drawn from this.

That's why I made my own table where I put information about each experiment:

• Is it on or off,
• Date of launch,
• Goal (what metrics I want to improve),
• Targeting (domain, group of pages...)
• Hypothesis (what I want to check),
• Result (which version won),
• Insight (what conclusions can be drawn from this).

It turned out very convenient. First, for myself – it motivates to organize work and write down conclusions, plus it is helpful to analyze the results. Secondly, for the team – when they see something strange on the site – they can look in the file and quickly understand, this is my experiment, or something broke :)

There is one thing that worries me. When the page is loaded, the original design is displayed and only a second later the changes of the experiment are applied. If I change the design at the bottom of the page - everything works perfectly, but if the changes relate to the title - the user notices a twitching.

Optimize recommends installing a plug-in that hides the page before it fully loads and the experiment changes are applied. But this delays loading of the page for a couple of seconds. And the slower it is loaded, the lower the conversion rate. So we decided to abandon the plugin and live with the twitching of the interface :(

Optimize is not perfect, and I use, probably, only 20% of its functionality but I have definitely started feeling confident, which is so necessary when your conversion rate depends on the weather conditions.

The more experiments you run, the better you understand the target audience and the market. You come up with more and more ideas and they are more and more successful. It's addictive :)

So, I think the tool is useful. Next steps will be to learn how to run experiments with dynamic content: cards of boats and popups. If you have such experience - I will be very glad to hear about it.