Holdout groups

You're running dozens of experiments each quarter. Your team is iterating faster than ever. But...

• How do you truly measure the cumulative impact of all these individual experiments?
• How do you quantify the overall ROI of your entire experimentation program?

This is where Holdout Groups come into play. A Holdout Group is a dedicated segment of your user base that is intentionally excluded from all experiments for a defined period.

This "control group" allows you to compare the performance of users who are exposed to your full experimentation program against those who are not, providing an unbiased view of the incremental value your experimentation efforts deliver.

Image source: Optimizely

Prove your impact

Holdout Groups don't just give you numbers; they give you narratives that resonate with leadership.

You can turn scattered test results into executive-ready business stories:

• Quantifiable ROI across your entire program, not just individual tests
• Clear evidence that faster iteration drives real business results
• Definitive proof that your experimentation investments pay off

No more "we think this helped." Now it's "we delivered this much revenue uplift this quarter."

Optimizely's approach allows you to measure and report on the cumulative impact of your testing program by withholding a percentage of your audience from testing.

This gives you a holistic picture of the overall long-term impact of potentially different experiments running simultaneously across your platform.

Why holdout groups are crucial for experimentation maturity

Four reasons:

1. Quantify true ROI: Individual experiments measure the impact of a specific change. Holdout Groups, however, enable you to measure the cumulative impact of all your experiments. This is vital for demonstrating the overall business value and ROI of your experimentation program to stakeholders and leadership.
2. Reduce bias and noise: Without a holdout group, it's challenging to isolate the true impact of your experimentation efforts from other factors like seasonality, market trends, or external campaigns. A dedicated holdout group acts as a clean control, minimizing confounding variables.
3. Build stakeholder trust: By providing clear, unbiased data on the cumulative impact of experimentation, you can build greater trust with product teams, executives, and other stakeholders. This fosters a culture of data-driven decision-making and continuous improvement.
4. Inform strategic investment: Understanding the true incremental value of your experimentation program helps you make informed decisions about future investments in tools, resources, and team expansion. It allows you to optimize your experimentation strategy for maximum impact.

Built-in, not bolted on

Unlike certain tools that force you to stitch together multiple tools and manual workarounds, Optimizely's Holdout Groups are seamlessly integrated into your existing workflow within your experimentation platform.

You can set up, manage, and analyze your holdouts exactly where you run your experiments without complex integrations.

What this means for you:

1. Seamless integration: No more manual workarounds or complex data stitching. Manage your holdouts where you manage your experiments.
2. Sharper insights: Gain clear, digestible insights into the overall uplift and value generated by your experimentation program.
3. Guided decision-making: No compromises on data quality or analysis depth. Your team can confidently iterate faster, launch features with greater assurance, and make smarter strategic choices.

By embracing Holdout Groups, organizations can move beyond simply running tests to truly understanding and optimizing the strategic impact of their entire experimentation ecosystem. It's a powerful step towards a more mature, data-driven culture.

Stay tuned for more details and best practices on leveraging Holdout Groups to scale your experimentation program!