Peakhour.IO - SEO

How To Exclude Query String Parameters from Search Engine crawling

2024-05-21T13:00:00+10:00

Last year we wrote about the problem of excessive crawling from search engine spiders. Search engines such as Google and Bing aim to index as much content as possible. For ecommerce sites, this often means indexing pages with query string parameters used for sorting, filtering, or pagination. Those parameters help users navigate the site, but they can cause a few predictable crawler problems:

Over-Crawling: Search engines may spend too much time crawling similar pages with different parameters, wasting crawl budget.
Duplicate Content: Pages with different parameters can be treated as duplicate content, weakening SEO performance.
Server Load: Excessive crawling can increase server load, slow down your site, and affect user experience. Search engines typically account for 30-50% of page requests to an ecommerce store. Managing their crawling effectively can have a material effect on site speed and server spend.

Another common cause of over crawling is internal searches being indexed.

In our previous article we mentioned using the webmaster tools provided by Google and Microsoft to manage crawler behaviour by adding ignored parameters. Since then, both tools have been updated and no longer allow you to add parameters to ignore during a crawl.

Differences in Crawling and Indexing

Search engines maintain an 'index' of web pages. Pages in this index are what appear in search results. To maintain the index, the search engine crawls a website to 'discover' new content and keep existing entries up to date. Webmasters can control what gets indexed with tags or headers in their web pages. These include:

Canonical Tags can be used to indicate the preferred version of a page. This helps consolidate link 'juice' and tell the search engine which URL to index.
Noindex tags can be used to prevent specific pages from being indexed. This is useful for thank you pages, admin pages or any content you don't want to appear in search results.
Nofollow links can be used to indicate to a search engine not to pass on SEO value to the linked page.

However, controlling what does or does not get indexed does not prevent content from being crawled. The only way to do that is via the robots.txt file. You may be familiar with the Disallow directive in the robots.txt file, but you can also use wildcards to prevent crawling of url parameters.

An example...

Consider an ecommerce store that has a category page which can then be customised with the following parameters:

    orderBy
    colors
    brands
    page
    results

These may appear in any order, and the combinations can result in 100s or even 1000s of variations of essentially the same page. Google is fairly smart when presented with this scenario, but Bing.... Bing can crawl very aggressively and it likes to try everything. In our example above, we may want to stop crawling everything except the page number, in which case an effective way to control crawler behaviour would be:

    User-agent: *
    Disallow: /*?*orderBy=*
    Disallow: /*?*colors=*
    Disallow: /*?*brands=*
    Disallow: /*?*results=*

We can't really do this in a single Disallow because the parameters might be in any order. By including the ? in the url we're ensuring that the parameter names are only in the query string, not in the main url path. This prevents crawlers from wasting crawl budget and putting unnecessary load on server resources.

Final Thoughts

Search engines can often make up 30-50% of the overall page requests to a website. Managing their behaviour helps maximise useful crawling and minimise server utilisation. Keep an eye on your access logs for unwanted behaviour, and use robots.txt where it gives you the right level of control.

Maximising Website Speed

2023-06-07T12:31:00+10:00

As businesses prepare for a global economic downturn, every source of friction matters. One of the most controllable is website speed.

For many customers, the website is where they first test whether a business is worth their time. They learn about the company, compare products, read content, and, if the experience holds up, buy. Loading time shapes that first impression, affects engagement, and can change whether a visitor becomes a customer.

This article looks at why speed deserves attention when trading conditions tighten. It covers search rankings, conversion impact, and published case studies where faster sites produced measurable gains.

The Need for Speed

Website speed is not an abstract technical score. It is how quickly users can see and interact with content. A delay measured in milliseconds can affect engagement, conversion rates, and customer retention.

Speed matters because user expectations are set by fast services and fast networks. When a page feels slow, people leave and are less likely to return.

Speed also affects how search engines, including Google, rank your website. For businesses trying to remain visible in a crowded market, especially during an economic downturn, performance is a practical lever.

Correlation with Search Rankings

The relationship between website speed and search rankings is supported by research and by statements from Google. A few years ago, Google announced that page speed would be a ranking factor. The change reflected Google's focus on relevant, usable pages.

Websites that meet all of Google's requirements receive a slight advantage, ranking one percentage point higher than the average. These requirements cover several areas, from content relevance and quality to mobile-friendliness and page speed.

By contrast, websites that fail to meet at least one of Google's requirements can sit at a measurable disadvantage, ranking 3.7 percentage points lower. That matters when search visibility is already under pressure.

Google's Core Web Vitals have also become a measurable factor in search rankings. These vitals measure aspects of page speed and user experience, showing how speed and SEO (Search Engine Optimisation) now overlap.

A study by Crystallize also found a correlation between speed and SEO. In their page speed score experiment, a page with a high score ranked #1 in Google with a featured snippet for the optimised item. Unoptimised pages with lower speed scores did not appear in search results.

The practical point is straightforward: website speed can improve search visibility. In an economic downturn, that extra visibility can matter.

Conversion Impact of Speed

Speed also affects conversion rates. Deloitte's 'Milliseconds Make Millions' report shows how small improvements in loading time can change commercial outcomes.

The study examined a 0.1 second decrease in loading time across different market sectors. In retail, a quicker page loading time led to an 8.4% rise in conversion rates and a 9.2% improvement in average shopping basket size. The travel sector saw a 10.1% increase in conversion rates and a 1.9% rise in average basket size. For luxury brands, faster loading times resulted in an 8.6% increase in page views per session and an 8.3% decrease in form bounce rates.

Peakhour clients have seen the same pattern. Pharmacy Direct reported a 30% increase in conversions and order value after reducing page load time by 90%. Kitchen Warehouse saw a 150% increase in revenue after decreasing page load times by 70%.

These numbers show that page speed is tied to business metrics, not just technical scores. The scale varies by site and sector, but the direction is consistent across the cited examples.

Real-Life Success Stories

The effects of website speed optimisation are visible in published case studies:

French linen brand Carré Blanc saw a 25% increase in conversion rates after improving web page loading speed.
Renault optimised the Largest Contentful Paint (LCP), leading to a 14 percentage point decrease in bounce rate and a 13% rise in conversions.
E-commerce platform eBay found that every 100ms improvement in search page loading time resulted in a 0.5% increase in additions to the shopping cart.
SnipesUSA.com doubled their average conversion rate from about 1% to about 2% by decreasing load times by 30%.
French toy retailer King Jouet enjoyed a 5% increase in conversion rates within a month through page speed optimisation.
AliExpress, a global online retail marketplace, experienced a 10.5% increase in orders and a 27% increase in conversions for new customers by reducing loading time by 36%.
Boutique designer brand Revelry saw 43% faster page loading, an 8% decrease in bounce rates, and a 30% increase in conversions after optimising images on their eCommerce site.
Zalando, an online fashion platform, reported a revenue increase of 0.7% per session by reducing web page loading time by 100ms.
Pinterest observed a 15% increase in platform registrations following an improvement in loading speed.
Telecommunications company Vodafone saw an 8% sales increase with a 31% improvement in Largest Contentful Paint ( LCP).
Mobile marketplace Swappie achieved a 42% increase in mobile revenue by focusing on Core Web Vitals.

These examples show how improving loading speed can lift conversion rates and revenue.

Optimising for Search Performance

Speed also affects search performance beyond organic ranking. Several examples point to paid search impact:

Lever Interactive Agency reported that one of their clients improved their Quality Score, resulting in a 17% decrease in Cost Per Click (CPC), a 31% decrease in Cost Per Acquisition (CPA), and a 20% increase in conversion rate on faster landing pages.
Crystallize Headless Commerce noted that scoring high in the Quality Score can lead to significant benefits, including up to a 50% discount on CPC prices. Conversely, a low Quality Score can result in paying up to 400% extra, severely impacting your marketing budget.

Core Web Vitals have also become a priority for eCommerce platform Shopify. The company continues to optimise speed performance to improve search rankings.

These cases show why performance work needs to be ongoing, especially where search traffic and paid acquisition costs are material to the business.

Enhancing Engagement

Engagement is not separate from speed. A fast, well-optimised site gives users less reason to leave and more opportunity to browse, compare, and interact. The data supports this in several ways.

Take eCommerce for instance. Customers are 10% more likely to recommend an eCommerce website when pages load in 10 seconds instead of 13 seconds. The likelihood of recommendation rises to 26% if loading time is reduced to 3 seconds. That shows how quickly performance changes user perception.

Other companies have also seen measurable effects from speed optimisation. Netflix implemented Gzip compression for resource optimisation, resulting in a 43% reduction in outbound traffic. Yahoo Japan News saw increases in both page views per session and session times (15% and 13% respectively), as well as a 1.72% decrease in bounce rate, by improving their Cumulative Layout Shift (CLS) by 0.2 points.

Google has also published data linking Core Web Vitals to engagement. Their data showed that favourable Core Web Vitals scores can reduce the likelihood of users abandoning a page before it loads by up to 24%. Meeting Core Web Vitals thresholds also led to an overall 43% improvement in bounce rate for The Economic Times.

The agriculture e-commerce platform, Agrofy, improved their Core Web Vitals scores by 70% for LCP and 72% for CLS, resulting in a 76% reduction in abandonment rate. Again, the useful lesson is not just that the site became faster. It is that users behaved differently once it did.

Key Speed Metrics

Website speed is about more than full-page load time. Several metrics help assess how fast and stable a page feels to a user. Google's Pagespeed Insights lists the following important metrics:

Largest Contentful Paint (LCP) measures the time taken to load the largest visible content on the page. The ideal target for this is less than 2.5 seconds. This metric matters because it provides a clear indicator of perceived load speed for the user.
Cumulative Layout Shift (CLS) evaluates the visual stability of a page during loading. The target here is less than 0.1. This helps limit content jumping or shifting while the page loads, providing a smoother user experience.
First Input Delay determines how quickly a page responds to user input, with the target being less than 0.1 seconds. This metric measures the interactivity and responsiveness of a website.

Together, these metrics show whether a website delivers a fast, smooth user experience.

User Expectations and Impact on Business

Users expect pages to respond quickly. When they do not, speed becomes a business issue rather than only an engineering issue.

According to Think with Google, slow-loading pages can affect user experience, resulting in higher bounce rates, negative brand perception, and an impact on conversions and revenue. When users have to wait too long for a webpage to load, they are likely to leave and look for a faster experience elsewhere.

Digital marketing expert Neil Patel highlights that a 1-second delay in page response can lead to a 7% reduction in conversions. To put that into perspective, if an e-commerce site is making $100,000 per day, a 1-second page delay could cost $2.5 million in lost sales every year.

Akamai also found that 53% of mobile site visitors will leave a page that takes longer than three seconds to load. This shows the standards modern users have for website performance and the revenue risk for businesses that fail to meet them.

Common Culprits

If your website is running slowly, a few common issues could be to blame. The usual causes are technical and operational:

Time to First Byte (TTFB) is the time it takes for the first byte of data to be received from the server. High TTFB can affect loading times and should be minimised.
Large Page Size and Resources can also contribute to slow loading times. This includes heavy content, such as images, videos, or large files. Optimising these resources can materially improve loading speed.
Third-Party Resources like ads, plugins, or widgets can require additional loading time. While these are often necessary, they need to be managed carefully to avoid excessive loading delays.
JavaScript can be a double-edged sword. While it enables advanced functionality, complex or poorly optimised JavaScript code can also hinder performance.
Single-Page Applications (SPAs) may experience slower initial loading due to their extensive scripting requirements, but they often offer faster navigation once loaded.
Busy Servers Handling Bot Traffic can also cause slowdowns. Bot traffic, in some instances, can account for over 40% of server load. Managing this effectively can help improve website speed.

Understanding which of these factors applies to your site helps you focus performance work where it will matter.

Continuous Monitoring and Performance Optimisation

Getting a site fast once is not enough. Speed can regress as content, third-party tags, releases, and traffic patterns change, so monitoring and performance optimisation need to be continuous.

Tools such as Google's Pagespeed Insights can help track website performance. Regular checks of key metrics can show which issues are slowing the site down and which changes need attention.

It is also important to test improvements on a staging website before deploying them to production. That reduces the risk of disrupting live performance or user experience. Regular diagnostic testing and iterative improvements help keep the site aligned with current performance expectations.

As SEO consulting company Moz highlights, focusing on continuous performance optimisation can have significant benefits. It can help maintain a fast, usable site and support higher search rankings, better engagement, and increased conversions and revenue.

Preparing for the Coming Recession

With an economic downturn on the horizon, a fast, well-optimised website becomes more important. Consumers are likely to be more selective with their spending, and businesses will need to compete harder for each sale.

A fast website can be a useful differentiator in this environment. It can boost your search rankings, making the site more visible to potential customers. It can improve engagement by giving visitors fewer reasons to leave. It can also increase conversion rates, which has a direct effect on sales.

In this context, website speed is not cosmetic. It is an operating requirement. The work is to measure the current experience, fix the main bottlenecks, and keep monitoring performance as the site changes.

The data and case studies point in the same direction: speed optimisation is a practical investment. It helps align the website with user expectations and makes the site a more effective part of the business.

Website speed is measurable, improvable, and commercially relevant. For businesses preparing for tighter conditions, it deserves active management rather than occasional clean-up.

When Bots Break Bad

2023-05-16T13:00:00+10:00

Bots account for a large share of web traffic. Recent studies put automated traffic at nearly 50% of all internet requests. Some bots are useful, such as search engine crawlers that index your site. Some are clearly harmful, such as scrapers and sneaker bots. Others sit in a grey area, including backlink and marketing bots from services such as Ahrefs and SEMrush. Even useful bots can create problems when they crawl too hard. This article looks at the main bot types and how to manage them with robots.txt and bot management tools.

Understanding the Different Types of Bots

'Good Bots'

Good bots perform legitimate work. Search engine crawlers like Googlebot and Bingbot index webpages so search results can stay current and relevant. Other examples include uptime and performance monitoring bots.

'Bad Bots'

Bad bots harm websites, users, or both. Common examples include:

Scraping content, copying and repurposing data from websites.
Sneaker bots, automatically purchasing limited-edition products (like sneakers) before human users can.
Spam bots, posting unsolicited messages and advertisements in comment sections or forums.
Vulnerability Scanners, trying thousands of website URLs to find security vulnerabilities.
Account Takeover, attempting to gain access to existing user/admin accounts using either credential stuffing or brute-force attacks.

'Grey Bots'

Grey bots sit between good and bad. They often serve a useful purpose and may follow crawling directives in robots.txt, but they can still cause problems when they crawl too aggressively. Common examples include:

AhrefsBot: A backlink analysis bot used by Ahrefs, an SEO tool.
SEMrushBot: A bot used by SEMrush, another popular SEO and digital marketing tool.
MJ12bot: A bot used by Majestic, a service that provides backlink data and analysis.
ScreamingFrog: An SEO analyser run from a local desktop.

When Grey bots (and even Good Bots) go bad.

Left unattended, grey bots can create practical problems:

Slow page loading times, which affect user experience.
Strain on server resources, potentially causing crashes, downtime, and higher costs.
Distorted website analytics, when bot traffic is mistaken for human traffic.

Managing Grey Bots with Robots.txt

The robots.txt file is a simple text file that tells web crawlers which parts of your site they can or cannot access. You can use it to manage bot behaviour and protect your website from aggressive crawling. Useful controls include:

Disallowing specific bots: You can block specific bots from accessing your site by adding a "User-agent" and "Disallow" directive to your robots.txt file. For example:

User-agent: AhrefsBot
Disallow: /

Limiting crawl rate: You can ask bots to slow down their crawling by adding a "Crawl-delay" directive:

User-agent: SEMrushBot
Crawl-delay: 10

Not all bots will follow robots.txt. ScreamingFrog, for example, can be instructed to ignore robots.txt and crawl a site as quickly as possible. You would not want a competitor doing this to your site.

Bot Management Tools

In addition to robots.txt, bot management tools (like those provided by Peakhour) can protect your website from abusive bots. Good bot management tools automatically block most unwanted traffic using a combination of Threat Intelligence, Fingerprinting techniques, Reverse DNS verification, and Header Inspection.

Advanced techniques like rate limiting and machine learning can help identify more sophisticated bad bots.

Search Bots and Double Crawling

Search bots like Bingbot can sometimes blindly follow links and crawl the same page multiple times due to different URL parameters. This double, triple, or worse crawling can increase server load and make indexing less efficient. eCommerce sites are especially exposed because product catalogues often have several filtering paths. We've seen Bing go haywire on a number of sites. Most recently, it was issuing around 50,000 requests per day to the search function of a Magento 2 store while cycling through parameters. This dropped to 2-3k requests per day when fixed. On another store, Bing was responsible for nearly half of all page requests (40k page requests) on a busy OpenCart store. Configuring it to ignore parameters dropped this to around 4k per day.

Configuring Search Bots to Ignore Query Parameters

Note: Since publishing both Google and Bing have removed the ability to ignore parameters when crawling via their webmaster/search console tools. See using robots.txt to instruct search engines to ignore query string parameters

To help search bots crawl your site efficiently, you can configure them to ignore specific query parameters. Use these methods:

Configuring Bing Webmaster Tools

Bing Webmaster Tools provides an option to specify URL parameters that should be ignored during the crawling process. To configure this setting, follow these steps:

Sign in to your Bing Webmaster Tools account and select the website you want to manage.
Navigate to the "Configure My Site" section and click on "URL Parameters."
Click on "Add Parameter" and enter the parameter name you want Bingbot to ignore.
Select "Ignore this parameter" from the dropdown menu and click on "Save."
Configuring Bing Webmaster Tools this way helps stop Bingbot double crawling pages with specific URL parameters, reducing server load and improving indexing efficiency.

Managing Other Search Bots

For other search engines like Google, use the relevant webmaster tools to manage URL parameters. In Google Search Console, follow these steps:

Sign in to your Google Search Console account and select the property you want to manage.
Navigate to the "Crawl" section and click on "URL Parameters."
Click on "Add Parameter" and enter the parameter name you want Googlebot to ignore.
Choose "No URLs" from the "Does this parameter change page content seen by the user?" dropdown menu.
Click on "Save."
Specifying the parameters you want search bots to ignore can prevent double crawling and make indexing more efficient.

Final Thoughts

When good or grey bots crawl too aggressively, they can cause the same operational problems as malicious bots: overloaded servers, slower pages, and worse user experience. Monitor website traffic and server load, set clear robots.txt rules, and use the major search engines' webmaster tools to control inefficient crawling. Done properly, this improves website performance and can lower infrastructure costs.

Locality Sensitive Hashing

2023-05-15T13:00:00+10:00

Locality Sensitive Hashing (LSH) is a computer science technique for designing hash functions for high-dimensional data. LSH aims to maximise the probability that similar items map to the same hash bucket. The principle is to amplify the difference between similar and dissimilar items, keeping similar items closer together while pushing dissimilar items farther apart. This gives a probabilistic approach to nearest neighbour search, a problem that appears in machine learning, data mining, and information retrieval.

Background and Usage of LSH

LSH was introduced by Piotr Indyk and Rajeev Motwani in 1998 to address the "curse of dimensionality," where traditional computational algorithms become significantly less efficient as the dimensionality of a problem increases. This makes LSH useful in contexts dealing with large and complex data sets, including text analysis, recommendation systems, image recognition, and genome sequencing.

The general idea is to use hashing to map high-dimensional data into a lower-dimensional hash space. The hashing scheme is designed so that collisions are much more likely for objects close to each other in the original space than for objects far apart. This supports sub-linear time complexity for nearest neighbour searches, which matters when working with large data sets.

LSH Variant for Load Balancing

LSH is also used in load balancing for distributed systems. A variant of LSH, known as Consistent Hashing, is often used for this purpose. In consistent hashing, the output range of a hash function is treated as a fixed circular space or "ring" (think of this as a circular hash table). Each node or server in the system is assigned a random value within this space, and each data item or request is assigned to the node closest to it in the hash space.

The practical advantage of consistent hashing is that when a node is added or removed, only an average of 1/n keys need to be remapped, where n is the number of servers. Conventional hash functions would require almost all keys to be remapped. This property makes consistent hashing useful in dynamic environments where the server set changes often, such as in a CDN (Content Delivery Network).

LSH in CDNs for Load Splitting and Caching

In a CDN, data is distributed across multiple servers in different geographical regions. The objective is to let users access the data they need as quickly as possible. This is achieved by routing each user's request to the server closest to them that has the data they want.

Consistent hashing, a variant of LSH, fits this scenario. By hashing both data and user requests, we can assign each request to the server closest in the hash space. This approach gives an even distribution of load across all servers, improving overall system performance.

LSH can also make CDN caching more efficient. When a request for a specific piece of content comes in, the CDN can use the hash function to find the server holding that content. The content can then be served quickly, improving the user experience.

LSH and Server Pool Rebalancing

LSH also helps with rebalancing server pools. In dynamic environments, workload changes constantly, and servers may be added or removed frequently. This can create an imbalance in load distribution across servers.

Consistent hashing mitigates this issue. As mentioned earlier, when a server is added or removed, consistent hashing only requires a minimal reassignment of keys. This keeps load redistribution to a minimum, so the system can quickly adapt while maintaining a balanced load across all servers.

Maglev and LSH

Maglev is a network load balancer developed by Google that uses a distinctive approach to connection distribution. It employs a variant of consistent hashing to balance load across a pool of servers. Using a large lookup table and a consistent hashing-like algorithm, Maglev can evenly distribute traffic across servers, even when servers are added or removed.

Maglev's algorithm starts by assigning a pseudorandom permutation of backend servers for each entry in the lookup table. These entries are then filled by iterating over the permutation and assigning each entry to a server. If a server becomes unavailable, its entries are reassigned to other servers using the same permutation. If a server becomes available, it is gradually reintroduced into the permutation, minimising disruption. This is a practical demonstration of LSH's principles applied to server load, high availability, and performance.

Conclusion

Locality Sensitive Hashing has applications from high-dimensional data analysis to load balancing in distributed systems. Its property of bringing similar items closer in the hash space and pushing dissimilar items farther apart makes it useful for handling large and complex datasets. In CDNs and server pool management, LSH, and its variant Consistent Hashing, support balanced load distribution and efficient caching, improving system performance and user experience. Its application in Google's Maglev shows how these principles can be used in production-scale load balancing.

Google AMP

2021-05-19T13:00:00+10:00

Google has announced that its long-maligned Accelerated Mobile Pages (AMP) platform is losing its preferential treatment in search results.

What is Google AMP?

Google introduced AMP as a way to provide optimised loading experiences on mobile devices. Google cached the content and served it from a Google domain name when it appeared in search results. Accelerated Mobile Pages ranked higher, and appeared in search results with a special badge indicating the page was fast loading, even if it didn't necessarily load faster than other sites. AMP was also a requirement for appearing in the Top Stories Carousel at the top of the results, or in Google News.

What is Changing?

Google is removing these advantages in search as part of the 'Page Experience Update', which is scheduled for June 2021. This update includes the much-blogged-about Web Vitals. Here's what Google said:

“The AMP format is no longer required and that any page, irrespective of its Core Web Vitals score or page experience status, will be eligible to appear in the Top Stories carousel. We're also bringing similar updates to the Google News app, a key destination for users around the world to get a comprehensive view of the important news of the day. As part of the page experience update, we're expanding the usage of non-AMP content to power the core experience on news.google.com and in the Google News app. Additionally, we will no longer show the AMP badge icon to indicate AMP content.”

Why is Google doing this?

AMP caused a large backlash from the internet community because many saw it as contrary to the idea of an open web. Google was inserting itself between the user and a website, and AMP introduced a number of headaches for webmasters. The official line is that with the new Web Vitals as ranking signals, Google didn’t want to overburden webmasters. However, given the control Google was exerting over websites with AMP, and the recent antitrust cases putting pressure on Google, it may simply have decided it was prudent to drop it.

What do I do now?

If you are currently using AMP, you can keep using it. If you aren't, you can concentrate on optimising your own website to create a good loading experience without using the Google-mandated way, and without being penalised. That's a good thing.

Conclusion

The imminent arrival of the 'Page Experience Update' means it's worth getting familiar with the Web Vitals and checking how Google views your site performance in the Chrome UX Report. And remember, Peakhour.io can make your website faster, more secure, and more reliable in under 5 minutes.

What is the Chrome UX Report (CrUX), and why should you care?

2021-02-26T13:00:00+11:00

A faster website is better for clients: they buy more, and they engage more with your content. However there's someone else that rewards fast websites: Google. Fast websites rank higher in organic search results than slower websites. They will also achieve higher quality scores in Google Ads, resulting in lower ad spend.

We've previously written about Google's Web Vitals. The practical question is where the real-world data comes from. It is not gathered by Googlebot. Google uses field data from the Chrome User Experience Report, usually shortened to CrUX, to show how eligible Chrome users actually experienced a page or origin.

That matters because a fast lab score is not the same thing as a fast customer experience. WebPageTest and Lighthouse help diagnose a controlled test run. CrUX shows the field data behind PageSpeed Insights, Search Console's Core Web Vitals report, and Peakhour's website speed comparison tool.

Introducing the Chrome UX Report (CRuX)

The CrUX report is a public data set of real-user measurements (RUM) of website performance across millions of sites. The report has been around since 2017 and is updated regularly, but the value is still often missed: it shows what real users experienced, not what a synthetic test predicted.

The data is collected from real Chrome browser users who have opted in to send browsing information back to Google. This opt-in requires that the user has:

Opted in to syncing browser history
Not set up a sync passphrase
Usage statistic reporting enabled

Despite these conditions, millions of Chrome users still report statistics back to Google. A given website still needs to be fairly busy before there are useful statistics in the report.

Gathered Metrics

The current Core Web Vitals are:

Largest Contentful Paint (LCP): how quickly the main content appears.
Interaction to Next Paint (INP): how responsive the page is to real user interactions.
Cumulative Layout Shift (CLS): how visually stable the page is while it loads.

CrUX also exposes supporting performance metrics and dimensions that help explain those headline scores. Older metrics such as First Input Delay (FID) still appear in older reports and tools, but INP is now the responsiveness metric to watch.

Dimensions

Because performance can vary widely, the metrics are divided into the following dimensions to help segment and understand the user experience.

Country
Device Type: Tablet, Phone, Desktop
Connection Speed: slow 2g, 2g, 3g, 4g, or offline

Viewing data in the report

There are several ways to see how your website performs in the report. These include:

Pagespeed insights

Google's website analysis tool provides summary CRuX data for the analysed URL and, if data is available, for the entire site.

Page Speed Insights Field Data" style="max-width: 100%;margin-bottom: 20px"/>

Google BigQuery

The most flexible option is to access it directly via BigQuery. You query it with SQL (database query language).

The downside is that you need to understand SQL and have a Google account.

Google's Search Console (formerly Webmaster Tools)

The search console now has a section 'Core Web Vitals' that shows whether URLs pass the Core Web Vitals, as well as a historical graph of performance for both mobile and desktop.

Looker Studio

Looker Studio, formerly Google Data Studio, can be used to build dashboards on top of CrUX data and other sources. It lets you visualise the performance of your website, or a competitor's website, over time.

Third party tools

Like our own website speed comparison tool. It uses the Chrome UX API to retrieve the information.

CrUX vs lab tools vs RUM

CrUX is field data. It is useful because it reflects real Chrome users, but it only reports where there is enough eligible traffic. It can also be slower to reveal the cause of a problem because it is aggregated.

Lab tools such as WebPageTest are better for diagnosis. They show waterfalls, redirects, blocked resources, caching issues, image weight, and third-party requests. Your own real user monitoring can go further again, because it can include business context that CrUX does not know: customer type, template, campaign, cache state, bot pressure, origin load, and release timing.

Conclusion - Why you should care

The data in the Chrome UX Report is one of the clearest public views of how Google sees the performance of your website. It is also a free source of real-world user measurements that helps you understand how visitors experience your pages.

Use CrUX to see whether your site is passing Core Web Vitals, use lab tools to find the technical cause, and use traffic visibility to understand whether bots, crawlers, bursts, cache misses, or expensive requests are putting the experience under pressure. If the first question is "how do we compare?", start with the website speed comparison tool. If the question is "what is slowing us down?", start with Traffic Management.

Core Web Vitals Optimisation

2020-09-11T13:00:00+10:00

A good user experience matters for any website. It matters to users, and Google also measures aspects of the user experience when ranking your website in its organic results.

Google ranks websites and pages in its search results using 'search signals'. These signals include the quality of content on a page, the number of sites linking to that page, and the experience of the user browsing the site. Factors that make up a great experience include ease of use, accessibility, speed and responsiveness.

The set of search signals that Google uses to measure user experience beyond a page's content value is called 'Page experience'. Google updated the Page Experience signals with metrics called Web Vitals.

For years, performance testing used different proxies to determine whether a website was fast. Browser events like 'Page Load' or 'Dom Load' were used before it was realised that these do not necessarily reflect what an end user experiences. Testing then fragmented, with different teams focusing on the measures they considered important. Web Vitals is Google's performance testing initiative to provide "unified guidance for quality signals that they believe are essential to delivering a great user experience on the web."

The 'Core' Web Vitals join the existing search signals: mobile-friendliness, safe-browsing, HTTPS, and no intrusive interstitials as seen in the graphic below from Google.

Understanding the Core Web Vitals

Before diving in, it is worth noting that the metrics that make up Core Web Vitals are expected to evolve over time. Google states that "these signals are not perfect and future improvements or additions should be expected."

Largest Contentful Paint (LCP)

Largest Contentful Paint (LCP) measures when the "largest", or "main" piece of content has completely loaded and is visible, usually a hero image. Content refers to text, foreground images, background images, and elements. LCP complements First Contentful Paint (FCP), a metric that marks the initial web page loading experience. LCP calculates how quickly a user can see page content. Scores below 2.5 seconds are considered in the 'Good' range.

First Input Delay (FID) (Due to be replaced March 2024 by Interaction to Next Paint (INP))

First Input Delay (FID) measures page interactivity: how long it takes for a page to respond to input from the user, such as a key press or a mouse click. Low FID scores ensure pages are usable. FID is a real-world metric that cannot be measured in the lab. The Total Blocking Time (TBT) metric found in Lighthouse is lab-measurable and correlates with FID to simulate real-world interactivity.

Cumulative Layout Shift (CLS)

Cumulative Layout Shift (CLS) measures visual stability while the page is loading. This means that once displayed, a piece of content stays where it is; it does not jump around the screen as other content loads. Lower CLS scores mean that users are not experiencing unnecessary content shifts. CLS scores below 0.10 are 'Good'.

Understanding the rest of the Web Vitals

The Web Vitals metrics that are not part of the Core Web Vitals are Time to First Byte (TTFB), and First Contentful Paint (FCP). They provide additional ways to improve a web user’s experience, and help diagnose specific issues, either in the lab or in the field.

Time to First Byte (TTFB)

Time to First Byte is the time it takes for a user's browser to receive the first byte of page content.

First Contentful Paint (FCP)

First Contentful Paint (FCP) "measures the point from when a web page starts loading to when ANY content starts rendering on screen." The term 'Content' means text, images, <svg> elements, or non-white <canvas> elements. Just remember that FCP can be triggered very early in the page load, but may not necessarily deliver any visible content or information to the user.

Conclusion

Web Vitals are Google's attempt to unify the metrics webmasters use to measure the Page Experience of their site. With Core Vitals included in the ranking signals, site owners can see which performance measures Google treats as important. Measuring performance is the topic of our next few posts.