Secrets for high performance web experimentation

From how to effectively monitor performance, defining benchmarks and how to leverage the latest Edge technologies to dramatically reduce latency with client-side experimentation. Below are the key takeaways.

So, why should you care about site performance?

Site performance encompasses page load times and the impact load speed have on key business metrics. 

Michael discussed the positive impact of faster load times on key business metrics, citing several studies conducted by Amazon, Google and many more. Each shows how positive key business metric performance, say conversion for Amazon or engagement with search engine pages for Google, is tied to site performance. Looking at site performance and key business metric impact, you can see the direct connection between milliseconds and dollars.

In addition to those key business metrics, those harder to quantify reasons also matter for site performance: customer satisfaction, user experience, and how people feel your site reflects on your business. If you have poor site performance, maybe a user will wait on that first visit but have already decided not to revisit such a painful experience! Gaining that understanding is where experimentation helps ensure you deliver the best digital experience to the user.

How is page loading time defined?

The key questions when measuring site performance: 

• How long does the initial page take to render? 
• What do people care most about on the page they’re loading? 
• What tools can be used to measure page load speed? 

Michael used the example of a COVID-19 tracker site. The page may well load right away, but the data you want arrives several seconds later. Which is the most honest way to measure page performance as it relates to the user experience and expectation? 

How can you monitor performance?

There are two ways to monitor performance: 

• Synthetic Monitoring generates fake internet traffic to your site, captures a variety of metrics as your page loads, and gives you an idea of your site’s actual performance profile. Because it’s synthetic, you get almost immediate feedback. Synthetic traffic, however, is not your visitors and does not represent their experiences on the site. 
• Real User Monitoring (RUM) captures and analyzes each transaction by users of a website or application. JavaScript code is injected on each page and reports on how long each page takes to load data for each request. 

RUM helps with understanding long-term trends, and Synthetic Monitoring helps diagnose and solve shorter-term performance problems.

How does experimentation affect performance? 

With all that on the importance of site performance, how can you add client-side experimentation and measure the performance impact? Thankfully, this can be done via a simple AA test, half with experimentation code and half without, then measure the impact on your business metrics. 

While the measurement is straightforward to configure, Optimizely has also released Performance Edge to help you keep website performance strong while incorporating client-side experimentation. Performance Edge dramatically reduces the overall performance impact of client side experimentation by intelligently deploying less code to every page – reducing the overall page load weight. 

So, how does Performance Edge work anyway?

Historically, Optimizely Web, for example, has used a single line of JavaScript which loads via a script tag. This file contains the data and the logic required to decide which experiments to execute on which pages for which users. It is loaded and executed in-browser before rendering starts. 

Performance Edge represents a whole new approach: the Javascript file is specifically created server-side to determine the experiments for a particular user on a specific page, so no code is included that isn’t pre-determined for this specific user. This vastly reduces the size of the payload returned to the browser, a fraction of the size of standard web platform products.

This effectively transfers work away from the browser on each page load having to load all potential experiments for an entire site and then decide which experiments are eligible for this user or which variation is required. Instead, a CDN layer – basically a bunch of fast, powerful server-side machines behind the scenes – figure it all out and send down this smaller, bespoke file.

Can Edge resolve flickering issues when asynchronous loading?

Performance Edge was built with reducing flicker in mind! Also known as the ‘flash of the unstyled content’ or the FOUC effect, that flicker is the result you get on some platforms that allow the original default content to render and then change it from the A version to the B version of the test. It’s not just a jarring experience, but one that can impact your results and lead to the wrong decisions being made.

Performance Edge means you don’t have to load asynchronously because the impact on performance is so small – in the low double-digit milliseconds range, compared to the hundreds of milliseconds or more of other platforms. 

Is Performance Edge available for server side experimentation? 

No – Performance Edge is specifically built for client-side testing. Feature Experimentation is our dedicated server-side product which already has the performance advantages of experimentation bucketing and execution occurring server-side. 

The caveat is that server-side experiments do not offer capabilities like the ‘what you see is what you get’ visual editor and some of these other usability tools that allow the broader team to run experiments outside of the confines of an engineering deploy schedule. So choosing what implementation to use will depend on the needs and structure of your team.