Preventing faceted navigation from generating unnecessary URLs

Decide which filter combinations deserve a stable, shareable URL before choosing the controls. Reserve URLs for combinations that make useful search landing pages, such as a meaningful brand-and-category or colour-and-size combination.

For routine browsing states, prevent URL creation instead of relying only on crawl or index controls. Faceted URL parameters can produce very large numbers of combinations, wasting crawler time on URLs that offer little value.

Map what the interface produces

On a representative category, record the address after applying one filter, combining filters, changing the sort and moving to a later results page. Look for parameters such as colour=red, size=10, order=price and page=2. Classify each pattern:

Use the classification to decide which states get distinct URLs. Review product-variant URLs separately: a variant identifying a particular item is not automatically a category filter.

Control creation, crawling and indexing

For browse-only selections, use client-side AJAX filtering to update results without creating a separate URL for every filter change. Keep distinct, useful filter combinations as URLs only when they need their own Google Search presence.

A URL fragment can represent a selection when that suits the experience, but Google ignores fragments. Do not use one to identify a distinct results page or a filtered page that needs its own Google Search presence.

If unnecessary filter URLs already exist, use robots.txt to block crawling of those URL patterns. This can reduce crawling, but it does not stop the interface generating URLs or guarantee a discovered URL will be absent from Search.

For an accessible page that should be excluded from Search, use noindex where Googlebot can crawl and read the directive. Add <meta name="robots" content="noindex"> in the page’s <head>, or return an X-Robots-Tag: noindex HTTP header; the header can also be used for non-HTML resources.

Google does not support noindex in robots.txt. A robots.txt block can prevent Googlebot from seeing a page’s noindex directive, so choose between limiting crawling and excluding an accessible page from the index.

Google Search Central’s “Pagination, incremental page loading, and their impact on Google Search” recommends distinct URLs and crawlable links for paginated results that should remain discoverable. Use a page parameter such as ?page=2, not a fragment, to identify a later results page; the guidance also recommends avoiding indexing filtered or differently sorted versions of the same results.

Preventing Unnecessary Faceted Navigation URLs – Key Actions

Check representative cases

Sample a useful subset, a browse-only combination, an empty result, a repeated filter, an out-of-range page number and a later results page. Compare each URL, visible products, HTTP status, robots treatment and product links with its intended role.

After adding noindex, use the URL Inspection tool to check the HTML Googlebot received. If the page still appears in results, request a recrawl; Google needs to crawl the page to see the directive.

Repeat the checks after a template or catalogue change, when a formerly useful combination may have become empty.

Validating Filter Combinations: Representative Case Checks