February 10, 2019
A/B testing in Google Optimize. Demigod level.
Olga Shavrina
For the last couple of months I have been learning a lot about marketplace A/B testing in Google Optimize, getting secrets from a neighboring startup product guru and finding several lifehacks which I will share with you with pleasure.
If you'd like to start from basic things – try the first post about how I became an addict of A/B testing.
If you'd like to start from basic things – try the first post about how I became an addict of A/B testing.
Dynamic elements testing
Popups etc.
It was tricky to find how to test popups and other dynamic elements. Say, I have a popover calendar and I want to play with its appearance.
In order to show a calendar you need to click a tiny grey icon in the top right corner which will switch the interface to an interactive mode, where you'll be able to click the field and a calendar will show up. After that you leave the interactive mode pressing an "Exit" button. The calendar stays on the page and now you can play with it.
It was tricky to find how to test popups and other dynamic elements. Say, I have a popover calendar and I want to play with its appearance.
In order to show a calendar you need to click a tiny grey icon in the top right corner which will switch the interface to an interactive mode, where you'll be able to click the field and a calendar will show up. After that you leave the interactive mode pressing an "Exit" button. The calendar stays on the page and now you can play with it.
Product cards
It was even trickier to find how to test dynamically loaded product cards. The essence of a marketplace's listing pages is to show a visitor relevant products as compact informative cards. The more attractive the card – the higher the probability a visitor will click it in order to see product details and order it. So the product card is one of the most important tools of a marketplace.
It was even trickier to find how to test dynamically loaded product cards. The essence of a marketplace's listing pages is to show a visitor relevant products as compact informative cards. The more attractive the card – the higher the probability a visitor will click it in order to see product details and order it. So the product card is one of the most important tools of a marketplace.
I use the term "product" in a wide sense. It can be anything: goods, cars or apartments for rent, language classes, freelance services etc. In our case it's boats for rent.
So, let's say I want to increase the size and change the text color of titles. If you select the title of one card and change it – the change will be applied to this card only. But you can have 10, 100, 1000... cards on a page and they change dynamically depending on the user's actions.
What to do?
Press Shift, select a couple of titles and you'll see the button "SELECT SIMILAR: 20" in the top left corner. Click it.
Press Shift, select a couple of titles and you'll see the button "SELECT SIMILAR: 20" in the top left corner. Click it.
After you configure styles, save the change. Then check, launch the experiment and you will see that all titles on the page have a new appearance. And not only the first 20 boats – if you upload more boats pressing a button "Show more boats" you will see all of them have a new title style applied.
Hide or not hide
Google Optimize recommends installing an Anti-flicker snippet in order to hide a page until the experiment's changes are applied.
I don't really like this idea, because with the plugin the page loading time increases dramatically and it is dangerous for the marketplace. That's why I came up with a trick :)
Users are used to seeing elements appearing on the page one by one. It's perfectly normal when the page is loading. But if an element disappears it can be confusing and doesn't look trustworthy. This is especially important for the elements in the header of the page.
That's why we do it like this – a new feature comes to production being hidden and then I show it with Google Optimize in one of the variants of the experiment.
Example
We decided to add labels "New boat", "Min fuel consumption" etc. to product cards. The frontend developer has added hidden divs to cards:
Google Optimize recommends installing an Anti-flicker snippet in order to hide a page until the experiment's changes are applied.
I don't really like this idea, because with the plugin the page loading time increases dramatically and it is dangerous for the marketplace. That's why I came up with a trick :)
Users are used to seeing elements appearing on the page one by one. It's perfectly normal when the page is loading. But if an element disappears it can be confusing and doesn't look trustworthy. This is especially important for the elements in the header of the page.
That's why we do it like this – a new feature comes to production being hidden and then I show it with Google Optimize in one of the variants of the experiment.
Example
We decided to add labels "New boat", "Min fuel consumption" etc. to product cards. The frontend developer has added hidden divs to cards:
My task is to find a hidden div in Google Optimize and make it visible.
<!–– Below is the hardest part, you can skip it and jump to the next section ––>
It's impossible to click on a hidden div in Google Optimize. That's why I examine the code and find a parent div that contains my div and calculate its sequence number.
I open Optimize and click any div close to the one I need. Then I move back along the line of breadcrumbs and find a parent div. It's super useful that it's highlighted on mouse hover like in a web browser's inspect mode.
I open Optimize and click any div close to the one I need. Then I move back along the line of breadcrumbs and find a parent div. It's super useful that it's highlighted on mouse hover like in a web browser's inspect mode.
Then I click the link MORE next to the parent div, get a list of child elements and select the one I need.
It's perfectly normal that the selection disappears at this moment because the div is hidden, and it's impossible to highlight a hidden thing.
Then we click an icon "Select" in the top left corner, see the popup and click "Add change" and select "Style".
It's perfectly normal that the selection disappears at this moment because the div is hidden, and it's impossible to highlight a hidden thing.
Then we click an icon "Select" in the top left corner, see the popup and click "Add change" and select "Style".
Are you still with me? Perfect! Let's go to the next step.
In order to show or hide an element I use an attribute "display". If an element is hidden the display attribute has a value "none", if visible – "block", "inline" etc. So in the "attribute" field of the next popup I write "display" and Optimize shows me the current value "none". Bingo! It means, I've found the correct div that's hidden so far.
In order to show or hide an element I use an attribute "display". If an element is hidden the display attribute has a value "none", if visible – "block", "inline" etc. So in the "attribute" field of the next popup I write "display" and Optimize shows me the current value "none". Bingo! It means, I've found the correct div that's hidden so far.
Then I change "none" to "block", click "APPLY" and see the pretty label in the product card. Yay!
Now we just need to repeat the same thing for another card, then choose both labels, press SELECT SIMILAR and apply "display: block" to all the elements selected in order to show labels in all cards of the page as I've described above.
Now we just need to repeat the same thing for another card, then choose both labels, press SELECT SIMILAR and apply "display: block" to all the elements selected in order to show labels in all cards of the page as I've described above.
<!–– The end of the hardest part. You can relax. ––>
Testing on several domains in parallel
If a marketplace supports several languages and you use different domains for this – it can be helpful for A/B testing.
I found out that A/B tests on different domains pretty often give different results, that's why it makes sense to test in parallel. It takes more time, but the result is worth the effort.
Why? At least three reasons:
I found out that A/B tests on different domains pretty often give different results, that's why it makes sense to test in parallel. It takes more time, but the result is worth the effort.
Why? At least three reasons:
1
Localisation. A simple translation error can influence a conversion rate and you'll never find it if you test only on one domain.
2
Different audiences. For example, organised, responsible Germans and frivolous emotional Italians have a way different behaviour.
3
Different content. Let's say, in France you have more products than in the Netherlands. You add a new filter that narrows down the selection of products. In France you can increase the conversion rate because users will be likely to choose the product from a relevantly small assortment and in the Netherlands you can drop the conversion rate down because users will frequently get an empty results page.
That is why parallel A / B tests in different domains give you not only more data to obtain statistically significant results, but also high-quality information that can give you more ideas.
A/A/B tests
At some point I started to doubt that tests are useful at all. Conversion rate is dancing, no variant wins, domains show different results. My trust in Google was jeopardized. And so I decided to run A/A/B test in order to test Google Optimize itself.
I've created a simple A/A/B test with one tiny change – put filter titles in bold and have run the test in two big domains – Spain and Germany.
I've created a simple A/A/B test with one tiny change – put filter titles in bold and have run the test in two big domains – Spain and Germany.
A/A/B-test on German domain
A/A/B-test on Spanish domain
Though the results are not statistically significant yet, both domains show that a new variant works better, that's good. The difference in probabilities of similar variants is 38-44% on the first domain which is acceptable, in the second so far 29-53%, which is slightly worrying and suggests that you can not 100% trust Google Optimize. I'm going to wait or a few more days in order to get more reliable results.
Test and test again
It never hurts to test and test again. Test all variants on all screen sizes, on all devices and web browsers (especially in Safari – it's extremely capricious).
In order to test a variant on Mobile – open a main page of an experiment on a Desktop, click the PREVIEW link and pick the "Share preview" option. Optimize will give you a link that you can paste in a web browser on Mobile and check your A/B test in a preview mode.
In order to test a variant on Mobile – open a main page of an experiment on a Desktop, click the PREVIEW link and pick the "Share preview" option. Optimize will give you a link that you can paste in a web browser on Mobile and check your A/B test in a preview mode.
Takeaways
Google Optimize is a cool and powerful tool. But! You can't trust it blindly. You better test and retest everything especially if you have a diverse audience, multi-language product, conversion rate dependent on weather conditions etc.
The most important thing is to treat the test results critically analyze and ask questions: what can be the reason of the difference in conversions on various domains; why did what seemed like a great idea not work; why does a solution work in all domains but one?
Does the brain blow up? Sure! But if you treat any result different than 100% positive, not as a defeat, but as a source of knowledge – tests become a tool of cognition and open up great opportunities for research.
The most important thing is to treat the test results critically analyze and ask questions: what can be the reason of the difference in conversions on various domains; why did what seemed like a great idea not work; why does a solution work in all domains but one?
Does the brain blow up? Sure! But if you treat any result different than 100% positive, not as a defeat, but as a source of knowledge – tests become a tool of cognition and open up great opportunities for research.
Olga Shavrina
Product manager. Human being