Lessons from Spotify - Scaling experimentation for growth

In this video Ashit discusses the value of UX analytics within experimentation programs and the importance of making use of collected data – also from inconclusive experiments.

He also shares how his experimentation team has grown and developed and how other companies might learn from the culture of experimentation they have established as the team is actually part of the business function at Spotify.


My name is Ashit Kumar. I work at Spotify and my job title is User Growth Lead. So we're at Opticon in Stockholm and I'm talking about the value of user experience analytics in a large velocity or high velocity experimentation program.

Make use of all collected data

I think when it comes to experimentation, generally, a lot of the experiments that you run are inconclusive. So generally in the industry, I've seen only one out of four experiments have like a clear result. But what happens to the rest of the experiments? You don't want all of that data, all of that knowledge to be lost.

And that's the point I'm trying to make to the audience today, essentially that they should make sure that all of the data they collect from their experiments is used in one way. And user experience analytics and using tools like Contentsquare, Hotjar, Clicktale, they can help them in this regard.

The team and how has it impacted the business

Our team was founded back in 2016, so it's our seventh year now, so it's been a long time. However, I think we have grown leaps and bounds since our team was formed. So when it was formed, this team was only based in Stockholm and it had one person initially at the start and now we have 13 people and we are distributed across five different locations. So Singapore, Stockholm, Barcelona, London and New York.

Our team runs more than 100 unique experiment ideas every year. So I would say in terms of impact, we have greatly helped our R&D teams to prioritize the correct features, the right features to work on. In that way we have really helped the user experience leaps and bounds. I wouldn't want to state like a dollar amount because in any kind of experimentation, right, there are more than one team that collaborate to make things happen. So I would say it has always been a joint effort and I think we have done our part in Spotify’s growth.

How democratized experimentation is at Spotify

Experimentation happens at every level at Spotify, right. So it's not just us, the dedicated CRO team that runs experiments. Every team sort of runs their own experimentation program. So for example, if there is a team that is designing a landing page, it's expected of them to test out different versions of landing pages before rolling one out. So I would say in terms of experimentation, we are fairly democratized. There are rarely like central functions, so our team is kind of an outlier in that regard because we are one of the very few teams that are dedicated to only run experiments. But overall, I would say at least 75 to 80% of teams at Spotify will do experimentation in one way or the other.

What other other companies can learn from Spotify

In terms of what other companies can learn from us, I feel like they can learn from the culture of experimentation because I feel like that's where we have been really successful. I've rarely seen decisions being made out of just like a HiPPO sort of thinking out loud and like: "Oh, I want to do this.". I think most companies should learn from making decisions from data. I think that's the first thing. And from the other perspective, even if you're not able to run scientifically valid experiments, there is still some value to be able to at least validate an idea. You just don't want to make decisions based upon one or few people's thinking. You just want your users to be able to tell you what they want. So it's more about being customer centric and it's more about and it's more about sort of applying data to decisions.

What is next 

I think the next step, as I said before, is to basically democratize the process of experimentation. Our team is set up somewhat differently as compared to other teams. We are actually part of the business function at Spotify, but we run experiments. So, we are not technically an R&D team. And what happens is a lot of our stakeholders are actually business stakeholders. They don't understand every aspect of CRO, or conversion rate optimization. So, what we are focusing on right now is to start working with some of these stakeholders as part of their team. So that just by working with us, they can learn bits and pieces of experimentation and develop a thinking of using data to make decisions.

And I feel like that is going to be our primary focus going towards next year as well, to do more of these embedding quarters. So we embed with one team for a quarter and then we plug out and then we actually see what kind of processes we can change by working with those teams.