A/B testing the whole funnel: visit, lead, payment with analytics in Tableau

We have started to save "A" and "B" values to the database, integrated the analytics system Tableau, learned how to track all the funnel: visit → lead → offer → payment, mastered how to run redirect A/B tests without page reloading and learned how to make cross-domain analytics.

I already wrote here and here that we love A/B testing a lot. And here I described our task prioritisation process and how we try to load the A/B test platform as much as possible because it's a bottleneck.

Let me remind you briefly. We – «Airbnb for boats» nautal.com. We have 9 domains: com, es, de, fr, ru, it, nl, br, gr. If you summarise all the traffic, you-ll get around 400k users in the low season, and about 1.2M in the high season. The business is seasonal, average check is above a thousand EUR, the conversion rate depends on the weather and cultural differences of users that's why A/B testing is fun :)

In Google Optimize you can use as a goal only website events. E.g. visiting the page or sending a request. But you can't keep track of the entire funnel up to the payment because users can pay a couple of weeks after they made a request, often from another device or using an offline bank transfer. Google Optimize cannot find out about it.

We run A/B test in Google Optimize. When a visitor opens the website, Google Optimise sets them a cookie with a parameter "A" or "B". If the user makes a request - we ask GA to give us the value and then save it to the Database.

As a result, we know how many requests we have with a parameter "A" and "B" so we can calculate a conversion rate, taking a number of visitors by variants from Google Optimize. There's one difficulty though – I have 9 domains instead of one so I need to run a separate A/B test for each domain.

As we have parameters "A" and "B" in the Database we can calculate any conversion rate we need – from requests to offers, from offers to bookings etc. for each variant. Google Optimize has nothing to do with it on this stage, hence it doesn't matter what web browser a client uses, do they have cookies or not and if they pay via bank transfer (what happens a lot because the average ticket is more than 1k EUR).

I can't just look into the Database, I need to ask developers to make a CSV extraction, upload it to Excel and then analyse it. It was super disturbing for developers until we came up with a solution to transfer the parameter to Tableau analytics system that we have recently integrated and started to use for product and business analytics. I'm so much in love with this system – a super flexible and powerful tool.

In Tableau I build a dashboard with a number of requests and bookings for every domain and A/B test variant for the period I need.

I still haven't figured out how to connect Tableau and Google Analytics, in order to get a number of visits. That's why I calculate the Visit → Request conversion rate in Excel by each domain, and then calculate total numbers for all the website.

Next step – testing the whole page layout. The last week we launched our first A/B test with two completely different designs of the boat page:

The original idea was to use URL redirect A/B test in Google Optimize. But we found out that in this case GA reloads the page that takes a couple of seconds. We couldn't afford it because as we are a marketplace – the page loading speed is critical for us and influences the conversion rate directly.

The second contraindication for using a redirect was SEO. Organic Google search is the main source of traffic for us, so we are super serious about SEO. When our SEO specialist discovered that I was thinking about a redirect A/B test, he got nervous, set hundred requirements and asked me to coordinate every step with him so that he could monitor the keywords' positions 24/7 and if something goes wrong, immediately turn off the test.

That's why we came up with...

...the idea to use Google Optimize only to split the traffic and instead of a making a redirect – change the page layout depending on cookies.

In order to do it, I create an experiment in Google Optimize with two variants "A" and "B" with identical content. I set the targeting to the boats' list page – it's the main landing page where we get traffic.

When the visitor enters the boat list page, GA sets a cookie "A" or "B" to their web browser. When the visitor clicks the boat card in order to open a boat page, we check the cookie and decide which layout we should use. If we see the cookie "A" – we show the old layout, if "B" – the new one. No reloading, everything is seamless. Voila!

When the user makes a request, we save the value "A" or "B" to the database in order to calculate all funnel conversion rates as it was described above.

Google Optimize turned out to be a valuable tool if you use it wisely. And in conjunction with Tableau and handy developers, you can work wonders with it. It's important to test every experiment and keep an eye on conversion rates because it's very easy to mess up and get a false result (or even to break the website).

In a year we advanced from "deploy and forget" approach to "constantly test on 7 domains, analyse the funnel from visit to booking" methodology. I don't even know what else to improve :)

The next step probably will be the mobile app A/B testing methodology. I have no idea where to start so far. But when we master it – I'll tell you all about it.