September 5, 2013

TechCrunch Interviews Optimizely President on His New Book, “A/B Testing”

Optimizely co-founder and President Pete Koomen digs into the contents of his new book, A/B Testing: The Most Powerful Way to Turn Clicks into Customers on TechCrunch TV.

Optimizely co-founder and President Pete Koomen digs into the contents of his new book,  A/B Testing: The Most Powerful Way to Turn Clicks into Customers on TechCrunch TV.

Full transcript:


Learn more about the book at

Colleen Taylor: Hi, you’re watching Techcrunch TV. My name is Colleen Taylor. Here with me in the studio is the co-founder and president of Optimizely, Pete Koomen. He, along with his Optimizely co-founder, Dan Siroker, just wrote a new book all about A/B testing. So, thank you for joining us here.

Pete Koomen: Thank you so much, Colleen.

Colleen: The name of the book here is, A/B Testing: The Most Powerful Way to Turn Clicks Into Customers.

Pete: Yep.

Colleen: Now, Optimizely is an A/B testing tool. Your entire company enables A/B testing, so why the need for a book when they could just go to

Pete: That’s a really good question. It would’ve been a lot less effort, too. The honest answer to that is that building great software that makes A/B testing really easy is only really half of the problem here. So, when we get started with a new customer, who maybe hasn’t had as much experience A/B testing before, one of the first things they ask us is, “Okay, this is great, but how do we actually approach this problem? What do we test? Organizationally, how do we structure this within our company?” Those are all questions that you can’t answer with software alone. So, we wanted to take a lot of the things that we’ve learned over the past four years and condense them into a resource that we could hand to new customers or anyone else who’s interested in learning about A/B testing so that they could start thinking about these questions, too.

Colleen: What I like about this book, since flipping through it a bit, is that it’s not just a theoretical textbook. You have a lot of real life examples in here. Can you talk about one of them? Companies use this, but also non-profits.

Pete: That’s right, yeah. The way we structured the book was we started with a bunch of high-level lessons that we’ve learned over the years about A/B testing, what typically works, some best practices. For each of those lessons, we tried to include examples of how this played out in real life. Tests that were very successful, even tests that weren’t successful, that help illustrate these lessons. Probably, my favorite example in the book is actually one that my co-founder and I were part of. That must have been three and one-half or four years ago, now.

So, the beginning of 2010, just weeks after we had decided to start Optimizely, we were contacted by the Clinton-Bush Haiti Foundation. They were raising money for earthquake victims. It had happened that week. It was a devastating earthquake. They had very quickly put up a webpage where they could take donations. The problem was that they had one very overworked IT person, who was spending most of their time focusing on just keeping the website up. Their traffic had gone through the roof. It was a huge operational job. They wanted help improving the performance of that website. So we buckled down. We spent three days with almost no sleep, and we ran test after test while building the Optimizely platform, and tried many different variations of that page. So, an A/B test is really just taking a webpage, trying several different versions of it, and showing one of those versions to each incoming visitor. Then, you measure which one’s more effective.

And so, what we did is we took the donation form and we added a photo of earthquake victims right next to the donation form. It turns out that more people were willing to go through the process of filling out that form when there was a reminder of why they were doing it. We tried removing fields. We tried changing the text size. We tried changing the text on buttons. And after those three days, we had a version that ended up performing 10% better in terms of dollars per eyeball on that page. And, that meant, over the course of the campaign, an extra one million dollars for earthquake victims. That’s my favorite example in the book.

Colleen: Great. One thing that I’ve heard from founders or different people in the tech industry is that Steve Jobs never A/B tested. Right? He just went with his gut. He had some vision for a design, this idea that people don’t know what they want until you give it to them, or someone with a great vision gives it to them. Is there an argument to be made that A/B testing isn’t necessary?

Pete: I think there’s an argument to be made that A/B testing isn’t the only thing you need, right? You can’t A/B test your way from a blank page to a successful business. It’s just absurd even to think about. But, when you think about what A/B testing is, it really is just a mechanism to get to understand your customers better, your visitors better. None of us is perfect, not even Steve Jobs. One of the ways you can account for that is by going out and talking to customers. When you have millions and millions of visitors, even thousands of visitors, it’s just not scalable to do that with everyone. And sometimes, the easiest way to understand how they’re going to react to something you want to do on your webpage, or a product that you’re developing, is just to try it out and test it. So, it really is just a feedback mechanism that companies can use to better understand their customers. And, it’s a very effective one. It’s not the only thing required. You need a lot of ingenuity. You need creativity and all the other amazing things that go into building something, but it’s a very valuable tool in that tool chest.

Colleen: Great. And, real quick, can we just hear a bit about where Optimizely is right now?

Pete: Sure. Yeah. We’re about three, almost four years old now, as a company, and we’ve had a great run. We just celebrated our 100th employee. We raised around, earlier this year, a led by Benchmark of $28 million, which was really wonderful for the company. We’re growing very, very quickly.

Colleen: Great. And now, you’re also a published author. Where can people buy this book?

Pete: People can buy this book at your local Amazon.

Colleen: Okay. [laughs]

Pete: Yes.

Colleen: A/B Testing by Pete Koomen and Dan Siroker. Thank you for coming by and showing us the book. Good luck. Keep us posted.

Pete: Thank you so much, Colleen.