Enabling experimentation at your organization: Determining your team structure

Scaling experimentation is often  the toughest challenge for organizations as they start to see the tangible results of experiments. Companies start to focus on how to make the practice a reality in all aspects of their business. Determining roles, responsibilities, and workflows of an expanding experimentation program may seem like a daunting task, but it is essential to making scaling a reality. 

Hopefully your organization is ready for scaling its program and success too! As the Lead Strategy Consultant at Optimizely I’ve  seen numerous ways organizations create this structure. There are typical structures experimentation programs use to ensure they can operationalize the practice effectively, and below I will outline which one may best fit your org And I’ll  walk through each of these structures and the benefits and drawbacks of each below:

• Center of Excellence
• Testing Council (Execution)
• Testing Council (Hybrid)
• Individual Team
• Hybrid

One thing I try to keep in mind and share with  customers is that these structures are just starting points. Every company  already has organizational structure and workflows to build their experiences and products within which experimentation needs to fit. These new structures should elicit an, “Ohh that structure looks familiar to what we are trying to do,” comment vs “We need to follow this structure to a tee.” That initial identified structure should serve as a  guiding light, but not the be-all-end-all.

A second item I often seek to drive  home is that the best experimentation programs have  a central team lead the program. Getting to a true culture of experimentation is not accomplished piecemeal. There has to be at least one person fully dedicated on a daily basis. And from there ideally more members of a central team with specific subject matter expertise. This cohort may not be fully dedicated to start, but it’s definitely something to strive for.

Here are three primary structures I’ve seen across best-in-class customers I’ve had the opportunity  to work with. Each can have a huge impact on an organization. Again, just as no two families are alike, so too are no two organizations. The only guiding rule here: you’ve got to make this unique to your org should you want to succeed. 

Center of Excellence

I hear this term the most when companies are looking to scale and create that culture of experimentation (it’s also a practice used heavily for organization’s analytics teams to make analysis on all digital work easier to access). But the definition I use for Center of Excellence when it comes to experimentation programs is a bit tighter than how most use it.

The organizations that employ a Center of Excellence are decentralized in their product and experience development. There are individual owners of those testing roadmaps aligned to the relevant products and experiences. Rarely do these overlap in ownership. This reduces the need to coordinate weekly on what testing is upcoming, prioritization, and steps to execute. 

A Center of Excellence is the model where you need a dedicated experimentation team – no questions about it. Without a central team to drive on-boarding and adoption of testing, the program can fail. Since the experimentation team won’t have ownership of product / experience to test on themselves they need to focus on being the experimentation facilitator such as sharing best practices, learnings, and measuring the output and impact of the program as a whole.

A Center of Excellence could be right for you if:

• Your organization already operates in a decentralized manner in other parts of the business such as analytics
• You don’t need individual teams to be in constant contact; they own their own roadmap so it’s better to not overburden them with constant review
• You’re more mature to testing – you have a strong understanding of what experimentation means to your company and each team understand methodologies 
• Having tighter centralization could inhibit velocity goals

My starting point to identify the structure with an organization is to write out where ownership is for the central team / leaders as opposed to individual teams. This can make it clear who owns execution for each step of an experiment’s lifecycle. 

Here’s what this looks like for a Center of Excellence (I have these in each of the following structures as well):

A good example of a Center of Excellence is Optimizely’s own product, design, and engineering organization. Our program manager Becca Bruggman’s main focus is facilitating the best practices, learnings, and education to our individual product managers. She’s in charge of setting the overall KPIs and OKRs of the program. Each quarter she reports back to leadership on our progress and identifies where we can spend more time in subsequent quarters to grow our program maturity. 

Testing Council (Execution Model)

What I see most companies employing today is some flavor of a Testing Council. I see two different approaches for this: Execution and Dispersed. The line between these two is where ownership lies when building and launching experiments. 

With both types of Testing Councils a weekly or bi-weekly meeting is a must to coordinate your prioritization and roadmap of tests. If you’re in an organization where different functions (i.e. marketing and product) or even individual product managers have shared ownerships of the experience being tested on, you’ll need to collaborate on an ongoing basis. 

With some Testing Councils I see a fully dedicated team managing it much like there’s a dedicated experimentation team in a Center of Excellence. What is unique is that a Testing Council’s dedicated team is an active participant in all aspects of an experiment’s lifecycle: helping ideate, test design, and analysis are all steps a Center of Excellence’s team rarely participates in. 

As the name indicates the Execution model has a central team that owns the execution (build and deploy) of experiments. Here’s a scenario: say it’s the end of your ongoing meeting and as a group you’ve determined what upcoming tests you are going to run and what you are going to do with the  results just analyzed. It is the central team’s responsibility to make that happen. You’re going to need dedicated development or engineering resources. It’s impactful because you’re only tasking the rest of the organization to focus on the highest value items: coming up with hypotheses and taking action on the program’s learnings. 

A Testing Council (Execution Model) could be right for you if:

• You don’t have full buy-in from development or engineering quite yet to execute experiments
• You do have resources in your central team to do experiment development; this may be more likely if you are using a client-side solution
• You want everyone in your organization to focus on contributing ideas and understanding learnings primarily

Testing Council (Dispersed Model)

On the inverse, the Dispersed flavor of a Testing Council puts the  execution step into the relevant development and engineering hands. This model fits when you don’t have those resources in the central team. Or if it’s easier to avoid collisions along the lines of  “what is this” moments if you allow the resources already building products and experiences to do the same for tests. 

A weekly or bi-weekly meeting is still necessary. Even though product or engineering may need to build tests it’s likely marketing, design, and other functions will have good ideas. And the more practitioners you have the greater the ability to grow the program. They already may be bringing these to product and engineering as changes that aren’t tests. I see this as an opportunity to teach them on why a hypothesis is useful for the changes they are requesting. Over time the consistent meeting may get less tactical and more on what’s upcoming and what results you’ve seen from tests. 

I often see this model when multiple product teams are experimenting on the same top-level domain. In retail this could be product managers who own individual steps towards purchase (Search, PDP, Cart, etc.). Traffic sharing, mutual exclusivity, or opportunities for multi-page tests can come up. 

A Testing Council (Dispersed Model) could be right for you if:

• You DO! have full buy in from development or engineering to execute experiments without their sign-off
• There’s a lot of cooks in the kitchen: many people are working on the same product and need to talk more closely about what is going to be done weekly
• Prioritization is a collective effort; you may have set criteria when someone submits an idea, but there’s a bit more “art” to the prioritization weekly
• You still want everyone to come together to review results even if execution and initial analysis is done in a decentralized way

Individual Team

I was in an Individual Team when I ran the program at the American Medical Association. It was just me and my team member from the analytics team participating and working in Optimizely. I had to be scrappy. I was fighting for opportunities to test. I didn’t have dedicated resources outside of our Digital Analytics team. I spent a lot of time educating folks and hoping to get to a  tipping point where leadership would say, “Let’s scale this.” I see a lot of companies still in this mode , so don’t be alarmed if this is where you are at. You have to start somewhere.

Since you may be feeling constraints on resourcing and where you can test you should be a little bit more creative with your prioritization. This could mean taking on ideas that may skip some of your standard scoring criteria when you can lean into other resources as they become available. And use those findings as a proving point (when you collect your results and learnings) to get more dedicated resources. 

An Individual Team is going to be a collection of folks who are only partially allocated to experimentation. But your point person should be at least 50% or  above. A lot goes into running experimentation effectively and you need someone spending a good chunk of their time on it. 

An Individual Team could be right for you if:

• You’re just starting out and have inconsistent involvement from product, development, or engineering
• You’re still teaching your organization what experimentation is and the value it can drive
• You want others in your organization to focus on providing ideas
• Where you can test is fairly limited at this point

On the other side of Optimizely’s business is our marketing team that owns optimizely.com and its experimentation. Today they operate as an individual team managing their own roadmap and executing experiments as they can within our broader initiatives on the experience. We still are working on how to best set measures for program output and better involve folks across our go-to market function. To their credit they’ve run some high learning experiments that informed our redesign earlier in the year and have dedicated development resources because of these  successes. 

Hybrid

For multi-tiered organizations with multiple products there most likely will be some combination of a Center of Excellence and Testing Councils (and maybe even Individual Teams). I call this model a Hybrid. A Hybrid will always have a Center of Excellence overseeing and supporting all testing teams and measuring the success of the program. A Hybrid could support any number of Testing Councils or Individual Teams. The number of those is not as important. 

Everything you’ve read above on the other models needs to be applied to the hierarchy below. You’re just taking the pieces of each and fitting them into your puzzle. Within the hierarchy you may need unique owners to run the individual Councils if you don’t want to burden the Center of Excellence with team to run those. But it may be best to have the Center of Excellence do just that. Since they are focused on sharing best practices and learnings, there’s no better way than living it with the sub-models each week. 

A Hybrid could be right for you if:

• You’ve got a lot of unique product and experience teams testing
• But there’s one (or maybe more!) product or experiences that need the collaboration of a Testing Council
• Your central team can support both being an internal facilitator of best practices and learnings, while ensuring the Testing Council runs smoothly
• You’re higher in maturity and have had a history of testing successfully on a few products or experience

The Importance of Some Centralization

As mentioned at the start, you’re going to need to enact a central owner of the program to make these models a reality. When establishing your central team you should focus on what the roles and responsibilities of the team will be for the day-to-day of the program. This central team will be the leaders of experimentation and should be given responsibilities that match just like any other of your cross-functional initiatives.

The ideal roles that should be represented in your central team are below. It’s rare to have a fully dedicated central experimentation team with all the roles listed below. These could be filled by having already established subject matter experts lend time to support the growth of the program. Other roles could also be introduced depending on what additional needs your program structure requires. 

A needed first step is to figure out where ownership lies for each of your central team’s roles. And where the others are needed to provide input. You can use a RASCI (or similar model) to determine these points. One thing to consider is that the below is just how Optimizely views the buckets of where ownership is needed. You should make these unique to your company and how experimentation will operate.   

As the central team starts to create governance plans, it should focus on these items highlighted in green below to ensure the other participants are focused on the highest value items in experimentation: ideation, test design, and analysis. 

I hope it’s abundantly clear that most organizations don’t have this level of detail at the start. And even rarer that the resourcing is there to match. But outlining a RASCI for what you have is still a useful exercise and can help identify those resourcing gaps you’ll need to fill. 

What to Think About As You Grow Your Program

Keep in mind that your structure may evolve over time. Perhaps in the near term you need to establish a variation of a Council to ensure everyone properly understands the experimentation methodology and how to properly build experiments. But over time you want to put more of that ownership into the individual team’s hands and develop a more traditional Center of Excellence. 

We see this often with our largest customers. We need everyone to understand the foundations of experimentation so we keep it a bit higher touch at the start. In parallel we can do more training to up-level those teams who are newer to experimentation. What we see to be the best training though is to 

——————–

Ready to get started with experimentation? Reach out to us today.

What team structures have you seen to be most effective in experimentation programs you have been a part of? What are some of the challenges to getting to a more decentralized structure? Leave a comment below!