How to build products your customers really want

In this recording you'll learn how experimentation turbocharges the efficiency of your teams in getting new products and features to market.

Today experimentation plays an essential role in accelerating digital product delivery, driving innovation, and ensuring experiences meet ever-changing user expectations.

We cover how experimentation can help you to:

• Prioritize your roadmap
• Mitigate risks when launching new features
• Promote your products to drive revenue

 

Transcript

Introduction

Hi, everyone. Welcome to our webinar on how to build products your customers really want. My name is Hiltsje Smilde.

I work as senior solutions architect at optimizely.

I work in our professional services team where I help all our customers to build optimizely the best way possible. I help with the implementation as well as with execution and QA of experiments.

Traditional product development challenges

Today, I want to share with you my experiment experience with helping product managers on how to build products your customers really want.

So first, let's have a look at this image.

It's probably a familiar image to you. Describing what can go wrong when developing software or products.

I always like the one in the top right corner. How the business consultant described it? That's usually how stuff gets sold. But what is the real product and what do customers really want?

But you can also see customers sometimes don't even know how to describe what they want because this one shows how the customer explained it in the top left. While on the bottom right, you can see what the user really needed. That's already a difference. And then there's all the different steps in between how the project leader understood it, how the analysts decide it, how the program eroded, what was documented, nothing.

That's a bit of a bad case, but usually documentation is less on the list. How it was supported, how customers were built, there's all this kind of differences between what a product is. And in the end, the customer just wanted this.

Often, the issues with software development and product development are coming from the traditional way of product developments with the waterfall method.

There is one person who listens to the customer, writes down some requirements, throws it down to the next layer who can then be a person who built some specifications.

The next person makes a design And in all these steps, you lose information. So by the time you actually launch your product to the customer, it's already not anymore what the customer wanted. There are huge delays and there's a big difference between what the customer wants and what you in the end and the building for them.

Agile product development

So one way of solving this is of course agile product development.

Almost everyone currently is working agile with the idea that making small iterations and asking for feedback in between launching small pieces of product along the way and getting real feedback from your customers is helping you to develop software and products better.

But how do you get this feedback from your customers?

Well, to really work agile and to get real customer feedback, We propose a way that's called experiment driven product development.

And with running experiments, feature tests, and launching your features via rollouts, you can really learn from real users what they want. And in that way, develop the best products. Every time you launch something, you get real feedback through an experiment or through a feature roll out to be truly an experiment driven product development.

Experiment-driven product development

So here you see some companies that are really good at this already. Companies like Facebook, Netflix, and Spotify are knowing for experimenting everywhere.

There is not a feature that gets launched immediately to all people. They first always test what is the best way to launch this feature? Or how should we change this in order for more people to use it? Facebook, for example, is known for launching things first in New Zealand to see how people there respond before rolling it out to the rest of the road.

This has been in the news a lot, so for example, Instagram stories was a new feature rollout.

You might have seen it that when you opened Instagram, you didn't have the stories yet, while some of your friends might have had the stories already available. They also use this to generate some need and some one for this. Oh, why do you have this? Oh, I want to have this too. So can really help you with marketing as well to use this kind of feature rollouts.

You probably also heard about Snapchat and just launching a new design without doing this roll out. They lost millions of users in one day. So using feature rollouts and using experimentation in your product development can really help you to mitigate risk and to not run into the same issues that Spotify had with losing a million millions of users in one day.

But how can you do this?

How do you integrate experimentation into your entire development life cycle.

Today, I want to share with you some examples, some of our customers and some of wisely ourselves because we use our own product on our own website as well to show you how you can use a experimentation in different, points of your development cycle to show this to share this with you to get some learnings from it.

Using experiments to improve customer experience

So how can experiments help you with before you build during launch and after you've launched a new product or a new feature.

So how can it help you in the discovery phase? How can we improve customer experience?

In the build phase, how do you develop and deploy features behind the feature flag? And how can it help you during roll out? How do you roll out safely and how do you then iterate on the feature to make it even better?

Prioritizing roadmap with experiments

The first thing I want to look at with you is how can experiments help you prioritize your roadmap?

One example of that is doing painted door tests.

A painted door test means that instead of building out an entire feature, you create the suggestion of a feature and see how many users want to use utilize it. You basically just create the notion of it without doing all the actual development work.

So one example of this on our own website.

In optimizely, you can click on create new to create either an AB test, a multivariate test or personalization campaign.

We added one extra option to it.

We had two variations of it. The one was called automated personalization alpha, deliver personalized content via machine learning, And machine learning, try it, deliver personalized campaign without manually targeting audiences.

So with this experiment or goal was to learn from our customers, what is a feature that they need? And what is the best way for us to learn about it. Now you understand that developing such an AI tool in the back end is a lot of work for our developers.

So we wanted to make sure that this is actually something our customers want before we would invest the time into actually building it. So we added these two options without making any backend. And when people would click on this, they would get the following popup.

Machine learning.

We're exploring personalization techniques based on machine learning and we're seeking early testers.

So we gave them an explanation of what we were planning to develop, and we wanted to hear from people if they would be interested.

If people click try it or, told us in another way that they want to use it, We would record that and we would check from our customers how many people would be interested.

Eventually, this made us decide that we wanted to develop this feature. And actually it just went live.

Another example is from our customer, the guardian.

Day questioned, would people use save for later functionality on the web?

The save related functionality was already, feature in their mobile application.

You see it here.

The first option they would have is to just build a save for later functionality for their website as well.

That would mean they would create parity between platforms, so people could say for later, both on mobile and on web.

But they have another option.

They could check with a pain in the door experiment if people actually want to use this save for later function. They wanted to measure the demand of this functionality because it would be quite some work for developers to actually create it and I wanted to know if there was demand for it before putting the work in there.

So they created a small button and whenever people would click it, That would get a popup again.

Saf for later isn't available yet on our desktop website, but we're thinking about adding it. Is this a feature you would use? And people could click yes or no. But with optimizely, they could measure how many people clicked yes and how many people clicked no. This was telling them if they needed to build this feature or no.

Now there are some things to be careful about.

Paint the door cautions.

Because creating such a painted door experiment is not always a very good user experience.

You can imagine if you would build five of this painted door experiments just on one page of your website and every other button someone clicks just shows a pop up, oh, we're developing this, which you maybe want to have this.

Then this will cause a lot of annoyance for your customers.

That's why we recommend to use paint the door test with caution.

Use it for things that you really want to develop. It might be a lot of work. Use it only like one time per page. And we usually also recommend to not rule this out to all of your users. You could, for example, choose to rule this out to only ten percent of users, which still gives you enough data about your customers, but also limits the negative impact on customers that it might have at the same time.

So what are some conclusions and action items from paint the door test and testing in the beginning of your product development life cycle.

First, Experiments can help you answer what to build and when to build. It might be the case that you learned from your painted door test that people are not currently interested in your cool new machine learning functionality.

It might be worth it to retry it again in six months because people are interested now. That's what I mean with when to build.

Challenge your team to test before building and to test during design.

Before building is a painted door test and during design is when you already start to develop it and you test different variations of it to make sure that eventually you're developing the right feature built in the right way.

A winning variation is not the same as a valuable experiment.

When testing in this starting phase of building a new feature, You will also find a lot of experiments that are losing or inconclusive.

This means that you learned something but you didn't have a winner.

So a winning variation is not the same as a valuable experiment because it's also valuable to learn that you don't need to build a feature. This will help you prioritize your road map better.

And testing during prioritization maximizes test value. Because testing this phase are usually not a lot of work to develop. It really helps you to prioritize your roadmap and maximize the value you get from your experiments.

So let's assume now we did obtain the door test. We found ourselves our customers really want this new feature that we just gave them the suggestion of.

How can we then proceed? We're building our feature but how can we then mitigate risk when launching that new feature?

Mitigating risk during feature launch

So we want to make always sure that when we launch something new, there's no controversy around it. We want to make sure people immediately understand how to use it. We want to mitigate risk during the launch of our new feature as much as possible.

One example of optimizely is our free trials.

So first optimize you had free trials.

Then at some point someone decided we're gonna remove free trials that didn't go well either, and then we decided, okay, We need free trials again, which that by then, because we had it removed was a new feature again.

And we need to launch this in a better way. So we want to have a way that we can control it, that we don't, like, immediately deletes all the codes when we side to remove it, but then we can have more control over turning it on or off or editing it in different ways.

So what did we do? We used a feature flag. So feature flag is a software development technique that lets you turn certain functionality on and off without deploying new code.

It allows for better control over your features during the life cycle.

You can toggle a feature off or on, and flags needed to be added to your feature code before launching with optimizing. So feature flags is basically a way in which you annotate your code and make it possible to without code deploy easily control these features.

You can then use a feature test.

In a feature test, you test different versions of your feature to see which one performs best.

You can also test the new feature against the current variance to understand the impact on your key metrics. So in this way, you can build your new feature, add it to a feature test, and then say, okay, how much added value does this feature have? You can also easily add feature variables to change the different features and to test different variations.

So back to the example of the trial length, We wanted to test what the best version of a free trial could be. We tested thirty days versus fourteen days.

And this is how that looks like in optimizely.

We had a variation called the thirty day free trial.

We turned the feature a new free free trial on, and we set the variable key trial length to thirty days.

And then we wanted another variation. The other variation was the fourteen day trial.

We also turned on the same feature, new free trial, and then set trial length variable to fourteen.

Without deploying code, it would now be possible to say, okay, fourteen and thirty both let's say they behave the same. We just wanna try out twenty one as well. Or, it was not it was too long. Now we want to make eight days.

We can easily do that as well. And we don't need to, all the time, go back to a developer to ask that. We can just decide that ourselves. We can just click and say now the free trial is this many days, and we can all the time keep testing new variations of this.

And it is possible to add even more variable keys to make your features really adjustable without going through code deployment ready. Again, that makes it possible to iterate really fast on the experiments that you are running.

Feature testing and control

We have some customer examples as well.

So we had a major department store looking to reduce the number of third party vendors.

They were unsatisfied with their recommendations provider.

So they used a full stack experiment to test an in house recommendations algorithm against the algorithm of their vendor.

So they created a feature that said new algorithm and tested that versus the current algorithm that they had from the fender. To compare the algorithms with each other.

Another example of experiments that you could do with this is what we call non negative experiments.

So with a non negative experiment, you want to do no harm to your most important metrics.

Sometimes you just need to launch a feature and you want to make sure it causes no harm to your most important business metrics.

An example of this is, for example, the cookie consent or GDPR banner that all websites now have.

This banner is a feature that you have to launch. But you want to launch it in a way that it does not negatively impact your business.

If you launch such a GDPR banner through a feature test, You can easily measure all the important business metrics such as revenue or conversion rate.

And keep track that days don't go down by adding this GDPR banner.

You can then also try out different versions of the banner that asks for consent from people to make sure it has the least amount of effect on your key business metrics.

This is another example of a customer.

So we had a major department store struggling to adapt to the digital age.

And they were thinking maybe it would be a good idea to add a revenue stream of advertising on their website.

Now there is a problem with ads on websites. It does not make them look prettier.

So they want to make sure that the ads don't distract people and make them click on the ad and go leave the website because that would cost them revenue on the normal add to cart and buying of products.

So they tested the new feature of ads and they tested it in a way.

How do we add this in a way that it does not affect any of our normal key business metrics. So no effect on add to cart rate, how many products people viewed, no effect on revenue, and things like that. So they wanted to place the ads in the most optimal way to get some revenue out of it, but not negatively impact the other revenue.

This is also a challenge that we usually see in media companies.

Media companies almost always have two different revenue streams.

So they have this revenue stream of ads and also the sign up for subscriptions.

With feature tests, you can balance the difference revenue streams and always make sure that the the one change does not negatively impact the other team's goals.

So let's say we did that feature test. We create a new feature, we launch it to a feature test, and we decided which one was best.

Rolling out new features

So once you know which experience is best, you roll it out to users in a controlled way. Because even if you've done the experiment and you know your feature is gonna be good, for users that are, coming back to your website a lot, They are used to using it a certain way. So if you change it a lot, they might get confused and leave your website instead of do what they're normally doing. So you always want to roll out new features in the most controlled way as possible.

So one way to do that is a staged rollout.

You start with ten percent of users and you slowly ramp up two hundred percent.

You monitor the impact and you reduce the risk for engineering also. There will be less load on the new feature. So if there's still anything they need to do in scaling on the back end, then that is something they can do while gradually rolling it out.

Something else you could do is a targeted rollout. Here you see the example that I used on Facebook in the beginning. Facebook likes to roll out things in New Zealand.

So what you do here is you target a roll out to a smaller market or Just one language to see how people respond there before you roll it out to additional markets.

Another step is a dark launch.

So what you can also do is that you widely certain people and they can start using it.

Then later, you activate the feature publicly without needing to code deploy because you first whitelisted people and later you remove the whitelist.

Another option to launch in this way is beta or exclusive access.

So just give certain people access to your feature.

So that is something that also really helps with the marketing of your new feature. This is something Instagram uses a lot. I talked about that in the beginning. They show certain new features, gifts, or stickers, or whatever, to to some influenced users, and they can use this new feature. This makes other people want it as well. And that's why it's really powerful to do this beta or exclusive access launches.

Make sure that you can test it on a small audience first and at the same time market the new feature, to a big audience.

One more thing you can do also is a gallery release.

To mitigate the risk you just send a small percentage of your traffic to the new release as a cannery build. It helps you mostly when discovering if something causes a load on your back end and on your service so you're sure you can scale it once you want to launch it to a big group of users.

The last one is a kill switch. This is something you have immediately built in when launching your experiments through optimizely.

As soon as you stop the experiment, no one would see it anymore. And this makes it easier to try out new features that you want to build that maybe your management team is not hundred percent sure about. It oftentimes helps to convince management of still trying out something If you can show them, look, here is one button. As soon as we get one complaint, we can immediately press this button and no one else will see this anymore.

This gives you as a product manager way more freedom to try out things that you think your customers might want.

And still make sure that you mitigate the risk by proving that you can easily turn it off when something goes wrong.

Conclusions and action items

So some conclusions and action items from rolling out your features and mitigating risk during launch.

So identify the core questions associated with a new feature.

So make sure that your new feature is measured in the right way, add the right metrics to your experiment to make sure that it influences the right metrics.

Turn those questions into testable variables during development.

So create different ways of testing it and measure your business metrics to make sure it does not negatively impact those business metrics.

We always say every launch should include an experiment.

When you're just launching features, you don't you're just guessing in the dark. You're just testing one version of it. That's why we say you should always test multiple variations, so every launch should include an experiment.

And last but not least, validate your assumptions with data.

A lot of times we think we know what our customers want and it appears quite often that that's not the case.

So it can be quite hard when you develop a feature and you think your customers really want it that in the end the data shows that they don't really care or it's not something that they wanted. Then you need to stop and iterate and go back and see where it went wrong because the data really tells you what your customers want.

Post-feature launch considerations

So let's say we did our feature test We decided which variation was best and we launched it. We rolled it out to our audience, but then we're not done.

So how can experiments help you to promote your products to drive revenue?

So when you develop your new feature and you know customers are interested in it, people are going to use it, but still there is returning visitors who might not know about the new feature. How do you convince them of using this? And also, you want people to actually make use more of your new products because in the end, you put a lot of work into it. So it would be nice if a lot of people actually use it. For this, you can use experiments to promote your product.

So some examples from Optimisely again.

This was created as an experiment a few years ago where we had just launched optimizely full stack. All our web customers would either get this popup showing them some information on our new full stack SDK or they would see some information on server side testing in this box. You see that's a favorite place for us to put some experiments in.

So people could either be triggered by server side testing through this box or to the pop up you previously saw. And in this way, we promoted our new full stack tool to our own customers.

Something else we use experiments for is for testing performance.

So This is when the Optimisely dashboard is loading.

And our hypothesis was is that people probably like using Optimisely more if the page loads faster.

And our engineers developed a new algorithm to load this faster.

Now you could say Wait, if your engineers know already that the other loading algorithm is more performant, why would you still want to do an experiment?

Well, Even if you think you developed something that's better, you still want to prove it and proving that you can only do while running an experiment.

So we decided to put this new performance algorithm in an experiment and we looked at the results and we had a metric looked at special execution time, and we actually improved the performance by over ten percent. Which was a statistical significant winner.

This means that now we can actually prove to our customers as well that by using this new algorithm, we improved performance by ten percent.

So I don't know if you can see it, but there's this little finger here in the corner that says view results.

I can imagine it's indeed not really clear.

All the other buttons of using optimizely are on the left.

But when you want to view the results of your experiment, which I think is quite important to our customers because you're not running experiments, just for running experiments, you're doing experiments because you want to get results. But the view results button was really hard to find for people.

So we decided to run an experiment to promote our results page.

And added an extra button where all the other controls are. And we measured how many people viewed the results page of their experiments.

And no surprise there. We increased page views.

Well, no. Actually, unique clicks on the few results buttons combined by twenty eight percent means twenty eight percent more people were able to see the results of their experiments, which is of course super important to us because we want people to see their results and then of course do something with it. So this was for us a way to promote a feature we already had for years, but we felt people should should use more often and we wanted to make it more easy for people to use.

Iterating on experiments

So some conclusions and actions items of keeping on iterating on your experiments.

So most important to remember experimenting isn't over when a new feature launches. So even after doing a feature test and deciding which version of your feature is best and rolling it out to your users, there's still more experiments you can do to make something even better for your customers.

Lightweight promotions provide meaningful insights.

These experiments don't always have to be big. You already did your big feature test. Now just use small changes such as adding a small button on the left so people can actually find it.

And last but not least, revisit launches to challenge conclusions and improve by iterating.

Even if you have decided after your feature test, a certain version is the best one, There's still room for improvement, and you can always go back and run another feature test again running with different variations.

In this way, you always keep iterating and keep improving your product for your customers. And that's how you build the product your customers actually want.

Conclusion and next steps

I want to thank you for your attention today.

And if you want to know more about product experimentation, We're running another series on product experimentation, which you can find through this link. Thank you very much.