I have sat through a version of the same review more times than I would like to admit. A team rewrites the titles on a category template. Three weeks later the slide shows clicks up 14% on those pages and the word "success" appears somewhere near the top. Nobody mentions that a core update finished rolling out in the second week, that the category in question is seasonal, or that the pages nobody touched went up 9% over the same period. The change may well have worked. The team has no way of knowing, and the slide has quietly converted a coincidence into a lesson that will shape the next six months of work.
That is the problem this article is about. SEO teams measure constantly. Measurement on its own does not tell you whether your work caused anything, and most of what passes for test and learn in SEO is a change followed by a look at Search Console. I want to set out how to do it properly, with the methods that are available for different kinds of change, what each one can and cannot establish, and how the whole thing fits into the way a team already plans its work. Some of this is uncomfortable, because done properly it produces fewer wins. It also produces decisions you can defend, which in my experience is what senior stakeholders actually want from an SEO function once the novelty of a traffic graph has worn off.
The working documents that go with this article, a test plan, a readiness checklist, a split test workbook, a single-page observation record, an AI visibility prompt log and a result report, are in the test and learn templates folder on Google Drive. Each one is linked from the section it belongs to.
Download the article
SEO Test and Learn as a PDF (9 pages, A4)
Keep a copy, print it or share it with your team. The test plan, checklist, workbooks and report are in a separate Drive folder, linked from each section and listed again at the end.
On this page
- Nobody knows exactly how search engines see their site
- Four methods and four levels of evidence
- A hypothesis is a prediction with a reason attached
- The control group tells you what would have happened anyway
- Most pages have no control, so the method changes
- Technical changes are validated first and measured second
- AI visibility is a distribution, so measure it like one
- How tests go wrong
- Rolling out on evidence
- Tests belong in sprint planning
- The purpose is a better next decision
- Templates and worksheets
- References
Nobody knows exactly how search engines see their site
Twenty-five years in this industry has taught me a great deal about how search engines work and almost nothing about how they will treat a specific website next month. Google documents its systems in some detail and still does not publish the weightings, the interactions between signals or the thresholds at which a page is judged good enough. AI search adds a second layer of opacity, because retrieval, source selection and the generated answer are each a process the publisher cannot inspect. On top of that, websites are not interchangeable. The same template change on two sites in the same sector can produce opposite results, because their architecture, authority, markets, languages and competitors differ. And whatever was true last year has a shelf life, since the systems change, competitors respond and customers ask different questions.
The practical conclusion is that an SEO team cannot run on accepted best practice and the instincts of its most experienced people, however good those instincts are. Mine have been wrong often enough to make me cautious about anyone's. The team has to build its own evidence about its own sites, and keep rebuilding it, because the evidence decays. That is what test and learn is for. It is how a search function reduces its own uncertainty rather than inheriting someone else's assumptions, and it is the mechanism that keeps the work pointed at the commercial objective instead of at whatever is easiest to change and measure.
Four methods and four levels of evidence
The first decision in any test is which method the change allows. Each one produces a different strength of evidence, and the report at the end should say which it was.
| Type of change | Method | What it can establish |
|---|---|---|
| A change to a group of similar pages (titles, a content block, internal linking rules, structured data on a template) | Split test: treatment and control page groups, served identically to users and crawlers | Whether the change caused a measurable effect, with a stated margin |
| A change to one important page or a handful of unlike pages | Directional observation: baseline, recrawl check, comparison with pre-period, seasonality and reference pages | Whether the result is consistent with the hypothesis; causation remains uncertain |
| A site-wide technical change (canonicals, hreflang, rendering, templates) | Validation first, phased rollout where safe, then monitoring of crawl, indexation and visibility | Whether the implementation is correct; wider effect observed over a longer period |
| A change aimed at AI search visibility | Repeated prompt measurement under fixed conditions, before and after, with reference prompts | Change in mention, citation and recommendation rates, with their spread |
A hypothesis is a prediction with a reason attached
Download: SEO Test Plan (Word)
Most SEO tests fail before they start because the hypothesis is a task description. "Improve category page titles" cannot be proven wrong, so it cannot teach anything. A usable hypothesis names the change, the pages, the metric, the expected size and direction of movement, the period, and the reason the team expects it. If we move the product noun ahead of the brand on the 96 category titles, non-brand click-through rate on those pages will rise by several points within six weeks, because the primary term is currently truncated in the result and users are not seeing it. That version can be wrong in four specific ways, and each way of being wrong tells you something.
The reason clause matters most and is the part teams skip. If the test fails, the reason is what you revisit. Perhaps the titles were not truncated after all, perhaps users were seeing the term and ignoring it, perhaps the pages never got recrawled. Without a stated reason, a negative result is just a disappointment. With one, it is a finding about how your site behaves, and that is worth more than most positive results, because it stops you rolling the same idea out across three thousand URLs on the strength of a hunch.
The control group tells you what would have happened anyway
Download: SEO Split Test Workbook (Excel)
Conventional A/B testing splits users. SEO testing cannot, because the thing being tested is how a crawler and a ranking system respond to a page, and every user and every bot should see the same page. What you can do, on a site with enough similar pages, is split the pages. One group gets the change. A comparable group does not. Both are served identically to everyone, so there is no cloaking question, and the untouched group becomes your estimate of what the changed group would have done if you had left it alone. SearchPilot and others have built platforms around this, and the method holds up without a platform if the site is large enough and the team is disciplined about the setup.
Comparable is the word that does the work. The two groups need the same template, a similar spread of traffic, similar query intent, no other changes planned during the test, and no structural link between them that would let the treatment leak into the control. Assign pages at random within traffic bands rather than by hand, because the hand will put the promising pages in the treatment group every time. Then do the one check that catches most bad tests before they begin. Compare the two groups against each other for the six to eight weeks before the change. If they move together, you have a control. If they diverge before you have touched anything, the groups were never comparable and the test will tell you nothing except that.
Here is the shape of it, with illustrative figures. A team selects 180 category pages on one template and randomly assigns 90 to each group, balanced by traffic. In the eight weeks before the change, weekly clicks on the two groups track within about 3% of each other, so the control is usable. Titles on the treatment group are changed on a Monday, recrawl is confirmed on a sample over the following ten days, and the team observes for six weeks. Treatment clicks are up 11% on the pre-period average. Control clicks are up 4%. The estimated effect of the change is therefore around 7%, which is clear of the 3% noise the pre-period showed, and the gap holds in five of the six weeks rather than spiking once. That is positive evidence. It is not proof, and it is only evidence about that template on that site, but it is enough to expand the test to a second group and, if it repeats, to roll it out.
Most enterprise B2B sites have perhaps two or three page families large enough to run this. Category and location pages often qualify. A hub of forty service pages does not, and pretending otherwise produces tests that look rigorous and mean nothing.
Most pages have no control, so the method changes
Download: Single-Page Observation Record (Excel)
The service page that matters most to the business is usually one of a kind. There is no second page like it to hold back, and the change is going to be made whether or not it can be tested cleanly. The honest approach is to accept that the evidence will be directional and to make it as good as directional evidence can be.
Start by writing down the normal swing. Pull eight to twelve weeks of weekly data for the page and the query group it targets, and note how much it moves from week to week when nobody is doing anything. If the usual swing is plus or minus 6%, a 5% rise after the change is noise and everyone should know that before the change is made. Then pick the primary metric to match the hypothesis, and add one guardrail metric that must not get worse, because a page can win on clicks and lose on conversions and the team should find that out from its own data rather than from the sales director.
| Hypothesis is about | Primary metric | Where it comes from | Guardrail to watch |
|---|---|---|---|
| Titles, meta descriptions, snippet presentation | Click-through rate on a named non-brand query group | Search Console, page and query filters | Impressions and position on the same group |
| Content depth, relevance, internal linking | Impressions and clicks on the target query group | Search Console | Enquiries or conversions from organic |
| Enquiries or revenue from a page | Qualified enquiries attributed to organic landing | Analytics and CRM | Clicks, so a drop in volume is not hidden by a rate |
| Indexation or crawl behaviour | Pages indexed, crawl frequency, valid annotations | Search Console, crawler, logs | Visibility on the affected pages |
Make one change, or if several are necessary, accept that you are testing the bundle and say so. Confirm the page has been recrawled before the clock starts; Search Console's URL inspection or the logs will tell you. Then observe for long enough, which for most pages means four to eight weeks and longer for low-traffic ones, and compare the result with the pre-period, with the same weeks last year if the page is seasonal, and with a handful of unaffected pages on the same site that act as a rough reference even though they are not a true control. Google's own guidance on diagnosing traffic changes lists the competing explanations worth checking, including updates, seasonality and technical faults, and that list is a decent discipline to apply to a positive result as well as a negative one.
A test run this way will often end with "probably" rather than "yes". That is the correct answer and the team should be comfortable writing it down. The point of the exercise is that the next decision about that page is made with a better understanding of how it behaves, which is more than the slide with the 14% ever provided.
Technical changes are validated first and measured second
Canonical corrections, hreflang, rendering fixes, structured data and template changes usually touch thousands of pages at once and often cannot be split without leaving part of the site broken. For these the first question is whether the implementation did what it was meant to do, and that is answerable within days. If the aim was to correct invalid hreflang clusters, success at this level is a crawl showing valid, reciprocal annotations and Search Console confirming the pages are indexed under the right canonicals. There is no reason to wait for traffic before declaring that part done, and no reason to call the traffic part done because the crawl is clean.
Where a technical change can be phased safely, phase it, and treat the early sections as the treatment group and the rest as a temporary control. Where it cannot, because the system is interdependent and a half-deployed state is worse than either whole state, deploy it fully, validate it fully, and then monitor crawl behaviour, indexation and visibility over a period that reflects the size of the change. Google is explicit that the effect of a change can take anything from hours to months to appear, and large structural changes sit at the slow end of that range. Much technical work is done to meet a standard the business has agreed to, and should be recorded as technically successful whether or not a traffic uplift can later be isolated. That is a legitimate outcome, and a team that refuses to report it will end up inventing uplifts to justify necessary work.
AI visibility is a distribution, so measure it like one
Download: AI Visibility Prompt Log (Excel)
Traditional search gives you clicks, impressions and position, and they are reasonably stable from one day to the next. AI answers are not. Ask the same question of the same system ten times and the brands mentioned, the sources cited and the description of your offering will vary, because the model samples. A single run tells you what happened once. A paper published in April 2026 by Schulte, Bleeker and Kaufmann makes the case plainly: visibility in AI search has to be treated as a distribution and measured repeatedly, because answers vary across runs, prompts and time. Anyone who has tried to track this by hand will recognise the problem.
The protocol that follows from that is not complicated, but it has to be followed.
- Fix a set of twenty to forty commercially relevant questions before any change is made, drawn from what customers actually ask rather than from what the team would like to rank for.
- Record the platform, market, language, logged-out state and location, because all of them change the answer.
- Run each prompt ten or more times per platform in each measurement window, and keep running them for several weeks before the change so the natural spread is known.
- Score three things separately, since they are different outcomes: whether the brand is mentioned, whether a page is cited, and whether the brand is recommended, which is not the same as being named.
- Make the change, confirm the updated content is accessible to the relevant crawlers, and continue under identical conditions, with a set of unchanged prompts or pages as a reference.
- Report the result as a rate with its spread. A mention rate that moves from 30% to 40% when the weekly spread is 15 points has not moved.
Google's Search Console now includes a generative AI performance report, introduced in June 2026 and available worldwide since the end of August. It is worth using and worth understanding for what it is: impressions only, by page, country, device and date, for Google's own AI features in Search and Discover. It carries no click or query data and says nothing about other platforms. It is a first-party signal that your pages are being surfaced in those features, which is useful, and it does not tell you whether a change caused that or what the answer said about you.
How tests go wrong
Download: SEO Test Readiness Checklist (Word)
I have made most of these mistakes myself, so the list is offered without superiority. Each one produces a confident conclusion. None of them produces a true one.
- Changing several things on the same pages in the same week and then crediting the one you liked.
- Choosing the metric after looking at the data, or sliding the date range until something turns green.
- Stopping the test the moment it looks positive, which guarantees that random highs get reported and random lows get waited out.
- Comparing against last year without checking whether the category is seasonal.
- Letting brand queries swamp a result that was supposed to be about non-brand demand.
- Reading a ranking change as a traffic change when the two have come apart, as they increasingly do.
- Starting a test in the fortnight either side of a confirmed core update and then arguing about which one caused the movement.
- Building a control group that the treatment pages link to heavily.
- The quiet one: pages dropping out of the index mid-test and being counted as a decline in the group they belonged to, when the real finding was that something broke.
Rolling out on evidence
Download: SEO Test Result Report (Word)
A single positive test is a reason to run a second one, and that is the discipline I would push hardest. Moving from one encouraging result on ninety pages to a site-wide deployment across every template is how teams turn a small win into a large regression. Where the type of change allows it, expand in stages: a small group, then a larger or more varied one, then full rollout with monitoring, and at each step keep a comparison alive so you can still see what the change is doing. Not every change can follow this path. Critical technical fixes go straight to full deployment, and some sites have nothing large enough to test. But for a title pattern, a content block, an internal linking rule or a structured data type, there is rarely a good reason to skip the second test, and a decision to do so should be recorded as a decision rather than slipped through.
The criteria for moving to the next stage are the same each time.
- The effect is in the expected direction and clear of the normal swing recorded before the change.
- It is consistent across consecutive weeks rather than a single spike.
- The guardrail metric has not moved against you.
- The implementation has been validated as working.
- For site-wide rollout, the result has repeated on a different group.
- The way to undo the change, and who decides to undo it, is written down before rollout.
Every test should end in one of four recorded outcomes, and the leadership job is to make all four acceptable to report.
| Outcome | What it means | Decision |
|---|---|---|
| Positive evidence | The result supports the hypothesis, with a stated strength | Expand to a second group, or roll out if already repeated |
| Negative evidence | The result suggests the change harmed the chosen outcome | Revert, or revise and test again |
| Inconclusive | The method could not separate the effect from normal variation | Extend the period, redesign the test, or stop and record why |
| Technically successful | The implementation passed validation; no uplift expected or none found | Retain where it meets the requirement, monitor where relevant, report it as such |
If people believe they must bring back a winner, they will find one, and the organisation will accumulate a library of success stories and no knowledge. A negative result that stops a change going across three thousand pages has paid for the whole programme.
Tests belong in sprint planning
None of this survives as an occasional initiative. It works when it is part of how the team already plans. In the operating model I have set out elsewhere, every sprint begins with a review of open tests before any new work is chosen: what are we testing, what has it shown so far, is there enough evidence to decide, do we extend, expand, roll out or stop, and does the result still serve the objective we agreed. Tests run across several sprints when they need to, and nobody forces a conclusion because the fortnight ended. The Global SEO Agile Playbook on this site has a template for the test record and a planning stage built around this review, so I will not repeat the mechanics here.
Two things are worth adding for international teams. A shared register of tests, successful or not, is the most valuable document a global SEO function owns, because an inconclusive result in one market saves four others from running the same thing badly. And a result does not transfer automatically. A title pattern that works in Germany is a hypothesis in France, not a finding, and treating it as one is a quieter version of the mistake this article opened with.
The purpose is a better next decision
It is tempting to judge a testing programme by its win rate, and that is the wrong measure. The purpose of test and learn in SEO is to reduce how much the team has to guess about systems it cannot see into, and to replace inherited assumptions with evidence about its own sites, markets and customers. That evidence points the work at the commercial objective, because once you know which changes move enquiries and which only move rankings, the backlog sorts itself. It also changes the quality of the conversation with the rest of the business. A team that can say "we tested it, here is what we found, and here is what we are doing next" is trusted in a way that a team with a graph is not.
The 14% slide will keep appearing. The job is to be the person in the room who asks what the untouched pages did.
Templates and worksheets
All documents in one folder
Open the SEO test and learn templates on Google Drive
Word and Excel files. Open the folder and use File, Download, or make a copy to your own Drive.
The documents follow the order of the article: plan the test, check it is ready to run, record it with the method that fits the change, and report the outcome.
- SEO Test Plan (Word). Hypothesis, pages, metric, expected movement, period and reason.
- SEO Test Readiness Checklist (Word). The checks that catch the mistakes in the section above before the test starts.
- SEO Split Test Workbook (Excel). Group assignment, pre-period comparison and the weekly treatment and control figures.
- Single-Page Observation Record (Excel). Normal swing, primary and guardrail metrics, recrawl date and reference pages for a page with no control.
- AI Visibility Prompt Log (Excel). Fixed prompts, conditions, repeated runs and mention, citation and recommendation rates with their spread.
- SEO Test Result Report (Word). One of the four recorded outcomes, the evidence behind it and the decision.
Related on this site: Why Agile Working Works So Well for Global SEO Teams, The Global SEO Agile Playbook and Agree the Commercial Mandate Before the Traffic Target.
References
- SearchPilot, The Math Behind SearchPilot: How SEO A/B Testing Actually Works. searchpilot.com
- Google Search Central, Website testing and Google Search. developers.google.com
- Google Search Central, Debugging drops in Google Search traffic. developers.google.com
- Google Search Central, SEO Starter Guide (on how long changes take to show). developers.google.com
- Julius Schulte, Malte Bleeker and Philipp Kaufmann, Don't Measure Once: Measuring Visibility in AI Search (GEO), arXiv, April 2026. arxiv.org
- Google Search Central Blog, Introducing Search Generative AI performance reports in Search Console, June 2026. developers.google.com