
Ecommerce Search
Part of Search result changes and AI features
Evaluating search feature changes with first-party performance data
Compare Search Console, analytics and business records to investigate search feature changes without claiming an unproven cause.
undefined
To investigate a search feature change, define the affected pages and the business action that matters. Compare the same reporting slices before and after the observation.
Search Console can show visibility and clicks; configured analytics and business records can show what happened after arrival. A before-and-after pattern is an association, not proof that the feature caused it.
Evaluate a search feature change with first-party data
- Define the page group and business actionStart with affected pages, not a site-wide total, and name the action that matters.
- Record the observation detailsNote observation date, queries checked, page URLs and site releases around that time.
- Choose comparable periodsUse equal-length, non-overlapping periods and keep Search type, country, device and page filters consistent.
- Locate the movementCompare the page group, then query, country and device; read impressions with clicks.
- Check other explanationsReview releases, redirects, indexing, offers, tracking changes, seasonality and a control group.
- Write a supportable conclusionName pages, filters, periods, observed change, alternatives checked and next action.
Define the question and comparison
Start with a page group rather than a site-wide total. For example: “Did our current service pages receive fewer relevant Google Search clicks after we first observed AI Overviews on these topics?”
Other search feature changes follow the same steps. A new filter or sort order on your site search, a changed snippet, or a reordered set of results can each be checked with the same page group, the same periods and the same filters.
Record the observation date, queries checked, page URLs and site releases around that time. An AI Overview seen once may not appear for another user or a later search.
Choose comparable periods of equal length that do not overlap, and keep the Search type, country, device and page filters consistent in both. Check those filters and definitions in the Search Console Help page Performance report (Search results): Overview and basic setup before you settle the slices.
An Australia country filter identifies where searches originated; it does not show whether each person was inside a local service boundary.
Use each measure for its own purpose
The Search Console Generative AI performance report is the first-party source for AI visibility on the pages and queries it covers. Read its help page, Generative AI performance report (Search), for what that report includes before you treat it as evidence.
Its coverage is not the same as the Performance report's, so check how each report defines and aggregates its metrics before combining totals. If the generative AI report is unavailable to you, record that as a data gap rather than inferring a cause.
Search Console Help also publishes Search generative AI control; note any setting shown there when you record what changed around the observation date.
Before treating an old address and its replacement as separate pages, check how the report assigns pages and whether redirects or canonical choices affect that mapping.
GA4 collects event-based data from websites and apps; a session is not a Search Console click. Compare the two periods in the [GA4] Traffic acquisition report, and keep GA4 as a separate comparison rather than treating its channel totals as equivalent, because the two sources use different definitions and need not produce matching totals.
Orders or enquiries may sit in the platform that runs your site, such as WooCommerce, Shopify or Magento, or in a separate CRM. Those records show whether actions met your criteria, but linking them to a search feature needs a supported attribution method.
Locate the movement
Compare the page group first, then identify the pages, queries, countries and devices associated with the change. Read impressions with clicks.
Falling clicks alongside falling impressions raises a different question from stable impressions and fewer clicks. Check query mix before treating an average rate as a verdict on the page.
Run the Generative AI performance report on the same page group for both periods. Reconcile its figures with the Performance report, chart and table views only after checking how each report defines and aggregates its metrics.
Check the time zone used by each data source and align periods before making a day-by-day comparison; an Australian business may record analytics in a local time zone.
Search Console data can be affected by anomalies, so check the Search Console Help page Data anomalies in Search Console when a surprising movement falls on an incident date.
Check other explanations
Review releases, redirects, indexing status, offers, promotions, tracking changes and seasonal demand, and compare the affected pages with a control group the change should not touch.
Check whether the measure was already moving before the change; a trend that predates the feature is not evidence for it. Inspect the affected page: does it still answer the query and allow a suitable visitor to act?
An analytics platform change is another explanation to rule out. Google Analytics Help documents that standard Universal Analytics properties stopped processing data on 1 July 2023, and that 360 Universal Analytics properties received a one-time processing extension ending on 1 July 2024, so a comparison spanning either date can reflect the platform rather than a search feature.
A useful conclusion names the pages, filters, periods, observed change, alternatives checked and next action. “Clicks fell on these pages; inspect their query mix and destinations” is supportable. A claim that AI Overviews took the site's leads needs stronger evidence.
Before concluding a search feature caused the change
- Release and redirect history reviewed
- Indexing status checked
- Offers, promotions and seasonal demand considered
- Tracking and analytics changes ruled out
- Control group compared
- Pre-existing trend checked
- Affected page still answers the query and allows action
- Data anomalies on incident dates checked
- Alternatives checked recorded in conclusion
Further reading: Search generative AI control - Send feedback on.



