Case study: how a fail, a survey and prototyping helped us to make a great report

This story will tell you how we've broken a Convead weekly report, learned many interesting things, made a user research and... whoa whoa let's start from the beginning.

During more than a year every Monday our customers received a weekly analytical report with main metrics, conversions, traffic channels and cities statistics and some more information. It looked like this:

We sent a report automatically every week but actually didn't know whether anybody read it or not because we sent it from backend and didn't see Open Rate and Click Rate statistics.

And overall we sent a lot of emails to our customers: onboarding, billing info, news ets, we began to worry if a report emails were excess and could they harm the other emails' Open Rate.

So the question was – leave the report of cut it off. We felt bad about cutting, so we decided to set up report sending through Convead itself (it has email-marketing module inside) in order to see statistics and make decent data-driven decisions.

We implemented it, tested, launched to production but... something went wrong and a report haven't been sent the next Monday :(

That morning we received dozens request from customers "Where is the report?", "What's happened?", "WTF!". The hypothesis that nobody needs a report was refuted. We returned everything as it was and decided to work on improving the report.

It's a pity we spent excess time for it. We could check the hypotheses quicker and cheaper.

Only customers could tell us how to improve a report so we decided to run a survey. It's ironic, but if there was no mistake we wouldn't know who to ask but thank to the mistake we got several dozens of users – hot and very interested in a report – those users who requested to technical support.

We come up with these questions for the survey:

Users were super interested in the report so we felt comfortable asking open-ended questions and it worked – we got many great insights.

We found out the most interesting things for customers are: number of orders and revenue in general and by individual channels and activities (who would have thought, huh?)

They use other analytics systems to compare data with previous periods. In our report, some numbers confuse them, especially abandoned carts count because they comprehend them not the way we planned and they didn't actually trust the revenue from Convead because it was not clear how it is built.

That were great insights for improving the report.

It's important to mention that we sent a report to all customers including those on a free plan. That's why improving a report we had to think not only about how to make customers happy but also about increasing sales.

So a report should show how great the customer business worked, engage a customer to use a service and encourage him to use paid features to increase his business metrics.

Convead is a B2B service but I always considered it as a friendly B2B. We are trying to be close to customers, talk casual language with them and be friendly. That's why we added encouraging phrases and emoji to the report.

We made a rough prototype in a couple of hours in a Google Docs file, shared it with several customers and asked for a feedback.

We got very positive feedback and several important comments. Approved a hypothesis that a comparison with a previous week is valuable.

We also figured out that traffic channel information is not detailed enough, we replaced a top-city list with a structured table, added more products to a "Bestsellers" block and improved its design.

A final report turned out to be twice bigger and 10 times more packed up with information than the first version.

In addition to beauty and informativeness, important things were added:

1. Recommendations for using Convead. E.g. if customer's abandoned view email campaigns are switched on – we show statistics, but if not – we write how much money he could earn if they had been switched on and show a call to action button "Launch".

2. Showed a value, a customer gets from the service – we made an accent on a revenue and orders, added a comparison with a previous week, highlighted revenue and orders Convead helped to bring.

The new report turned out to be much better than the old one - more informative, unambiguous, it gives an idea of the business dynamics, shows how much profit traffic channels and marketing activities have brought, and also helps us to involve and retain our customers.

Most likely, we need to make another iteration in a while, conduct a new poll and interviews, throw out less popular blocks, rearrange something or add new data. But for now, we are happy with the result and can move to more priority tasks :)