Measure twice, cut once
Web Performance Testing Tools and Tips
Many organizations struggle with site load times and have yet to adopt the right measurement tools or processes to improve it. As the product manager for Optimizely Web, naturally, I’m passionate about web page performance because it’s a major influence on the user experience and your business goals. To help our customers deliver snappy experiences for their end-users, we recently launched Performance Edge. It enables performant experimentation at scale, by reducing the impact on site speed. If you are ready to supercharge experiments on your most performance-sensitive pages with Optimizely, or you are starting to think about setting site speed goals for the year, following the best practices below will help your team improve performance scientifically.
Prioritize Performance. Latency negatively impacts the user experience. Your site’s KPIs are a function of the user experience. It’s as simple as that.
Measure the right numbers. Many teams struggle to measure performance accurately. They lack focus when it comes to the metrics that matter. These metrics indicate how your performance is shaping the user experience. For example, Time to First Contentful Paint (FP/FCP) will let you measure the time it takes for a user to see something material on the page. It detects when the first major visible element, such as a hero image, renders. FCP is a pretty standard metric, and many tools like webpagetest support this out of the box.
Better yet, measure Time to First Meaningful Paint (FMP). FMP captures how long it takes a meaningful element to load. It’s up to you to decide which element makes the paint meaningful for your users. At Optimizely, I work on A/B testing products that modify web elements as the browser loads them. The element tested in variations of a given experiment, is the one that we consider meaningful. Time to Interactive (TTI) is when a visitor can click or tap and is important overall page performance health. Still, given that it is the outcome of even more resources loading in the browser than other metrics mentioned, it is less useful for identifying specific actions you can take to improve.
Use the right tools. Synthetic testing tools (with network throttling to mimic mobile) help give you an initial read, but there is no substitute for real-world traffic. Using real traffic is called Real User Monitoring (RUM). It is important to make sure your RUM collects info like the visitor’s browser/device and location so you can slice your data later (more on that below). Synthetic tools work too, sometimes letting you mimic mobile traffic, but they usually suffer from a limited sample size issue.
Use the right analysis technique. Performance data is involved. There can be lots of variance and outliers. Visitors’ devices and locations are literally all over the map. Performance timings tend to be unstable over time. Most sites are built on or with dozens of 3rd party technologies like CDNs, frontend frameworks, A/B testing tools, databases, and APIs, to name a few. Your data is unlikely to reflect a perfect bell-curve. That is, it’s probably not normally distributed and will have a long tail due to outliers.
The best way to analyze website performance in the face of noise is to segment your visitors’ requests, measure it over time to account for seasonality, examine a large sample size, and use percentiles. Using averages instead will cloud your understanding because a small number of hanging requests (due to things like a CDN cache miss or spotty connection) will move the average…towards the outliers.
Run proper tests. When you want to improve your site’s performance, a) not everything is a silver-bullet, and b) you should measure and communicate the impact of the changes you’re making. Testing helps quantify any tradeoffs you’re making and enables you to communicate the impact of your work.
This Performance White Paper covers many of the performance testing topics in this post. It expands on how traditional client-side A/B testing tools impact your site performance, and how Optimizely can help ensure that the value of your experiments is higher than the performance cost of delivering them.