Challenge: Answering Leon's Questions

Leon, one of the stakeholders for Johanna’s projects, wants to be sure that they’re tracking revenue to prove the impact of her experimentation on the company’s bottom line. He’s asked Johanna for a quick face-to-face meeting to help him understand how Attic and Button can use revenue tracking to show the impact experimentation is having on revenue. Why don’t you try sitting in Johanna’s chair and see if you can field Leon’s questions?

Thanks for meeting with me. I want to be sure this experimentation actually makes the splash we claim it’s making. You said in your last team meeting that revenue can’t be our primary metric in experimenting. I don’t understand that. Our primary goal as a business is revenue; why can’t we make that our experimentation goal?

  • There’s no reason we can’t make it a primary goal of an experiment, as long as we make changes and have variations that are clearly defined. With Optimizely’s revenue tracker, we can easily monitor how our experiments impact our bottom line and give you the metrics you’re looking for.

  • We can’t make it a primary metric for experimentation because it’s not tied directly to an action our users take on a page. We need to tie experiment goals to changes we can directly measure. But we can set up revenue tracking to track revenue as a secondary metric.

  • That’s not a problem. We can easily make revenue our primary metric by using the revenue tracker. It will allow us to see exactly the impact that our experimentation is making on revenue.

I’ve heard a little bit about the revenue tracker, but don’t know much about how it works. What exactly does it do?

  • Revenue tracking will help us by highlighting our website purchases in our analytics to help us more easily see how our overall revenue was impacted by our web sales.

  • We can use revenue tracking to help us figure out exactly how our experiments impact the bottom line.

  • Revenue tracking will allow us to see our overall revenue changes from our website. We can be confident that any revenue changes that happened during the experiment period are based on the experiment.
  • Revenue tracking can track total revenue from purchases or other events on our site and attribute the amount to a specific experiment and variation.

How will we be able to track and categorize our revenue for analysis?

  • Revenue tracking allows us to see our revenue correlated with our custom segmentation. We can decide which segments you want to see and how you want to set up tracking when we initiate our experiment.
  • With revenue tracking, we’ll be able to see the revenue earned per visitor, revenue per paying visitor, purchases, and the total revenue earned per variation on our results page.
  • We can leverage a separate revenue tracking software to give us even more specific analyses of the revenue changes as a result of this experiment. Finding secondary software will allow us to be able to customize exactly what we want to track.
  • With revenue tracking, we’ll be able to see the revenue per visitor broken down by order number and date on the results page.