Across my time at Iron Mountain, at Staysure and running RichWeb Media I have been responsible for organic search on estates of very different shapes. At Iron Mountain that meant more than 60 sites in 35 languages. Multiply those together and you have more template and locale combinations than any team will ever check by hand, served by several engineering teams on their own release schedules, and measured through a Search Console view that arrives two days late. That arithmetic is the whole argument for this article.

On this page

  1. The estate outgrew the audit
  2. Why the tools matter more now than they did
  3. Where automation earns its place
  4. I have built versions of this before
  5. What the SEO team should hand to a machine
  6. What the technical SEO team should build
  7. Developers need checks that run in minutes
  8. Where trained models help and where they do not
  9. The measure of the system
  10. Sources

The estate outgrew the audit

The traditional answer to an estate like that is the audit. A crawl runs, a report lands, a backlog gets built and prioritised, and a quarter later the exercise repeats. The audit is still useful. It is also a photograph of a river. By the time anyone reads it, the release that introduced the problem has shipped, the next three releases have shipped on top of it, and the person who wrote the offending template change has moved to another ticket.

What a large estate actually needs is something that notices the problem on the day it happens, ideally on the day before, while the change is still in a branch and costs nothing to fix. That is what technical SEO tooling is for, and it is why I think building it is now part of the job rather than a side project for the people who happen to enjoy code.

Why the tools matter more now than they did

Four things have changed since the audit was good enough, and they compound.

The first is release cadence. Most large sites now deploy daily or several times a day. Every deployment is a fresh chance to change a canonical, lose a hreflang set, flip a template to noindex or point a redirect at the wrong place. When DeepCrawl (now Lumar) ran its Automator tool in beta in 2020 it reported SEO defects in roughly 35 per cent of releases tested. I have no reason to think that figure has improved since, because the number of releases has only gone up.

The second is the number of hands on the site. I saw the same pattern in every one of those roles. The content team, the product team and several development teams all change the same pages, often without knowing what the others shipped that week, and often from different offices in different time zones. Each team runs its own backlog and its own definition of done, and none of those definitions mention organic search. The complexity rises fast from there. Add multiple countries, languages, servers and websites, sometimes a portfolio of brands each with its own stack and its own developers, and the same failure modes appear in a dozen places at once, introduced by people who have never met. A check that lives in one person's head protects one site on one team's release. A check that lives in a shared config protects all of them, whoever made the change.

The third is rendering. More of the content that matters is assembled in the browser, which means the HTML a crawler receives and the page a user sees can quietly diverge. Nobody notices until a section drops out of the index, and by then the diagnosis is a week of work.

The fourth is AI search, and it raises the stakes on all of the above. Answer engines and generated overviews have changed how traffic arrives, but they have changed nothing about the plumbing underneath. A page that cannot be crawled, rendered and understood cannot be cited by a model any more than it could be ranked by a classic index. Two things about how these systems work make the technical layer matter more rather than less. The first is query fan out. A single question from a user is expanded into many related queries behind the scenes, each of which retrieves its own candidates, so a site has far more chances to be considered and far more chances to be found wanting on a technical detail. The second is speed. These engines frequently fetch pages live while composing an answer, and they work to tight time budgets. A page that is slow at initial load is simply skipped in favour of one that responds, however good its content is. The consequence runs in a straight line. Fewer fetches means fewer mentions, quotes and citations, which means less brand visibility in the answers people now read instead of clicking, which means fewer referrals and lost revenue. Performance used to be a ranking factor among many. It is now closer to a gate on whether you are in the conversation at all.

Put those together and the cost of a single missed technical defect is now higher, the number of opportunities to introduce one is higher, and the window to catch it by reading a report is shorter. Tooling closes that gap. Nothing else does.

Where automation earns its place

Not everything in SEO should be automated, and the teams that try usually end up with a dashboard nobody opens. The useful question is which kinds of work a machine does better than a person, and in my experience there are four.

Repetition. Any task that runs the same way every week is a candidate. Pulling Search Console exports, diffing this week's crawl against last week's, checking that every sitemap URL still returns 200, validating that hreflang pairs still point at each other. People do this work badly because it is boring, and boring work is where errors hide. A script does it the same way every time and never skips a locale because it was Friday afternoon.

Prevention. This is the highest value area and the least practised. The cheapest moment to fix a technical defect is before it ships, when the change is still in a pull request and the developer who wrote it is still looking at it. Checks that run at that point, fast enough that nobody minds them, stop a whole class of incidents from ever reaching the index. I cover what those checks look like in the developer section below.

Early warning. Some defects will get through regardless, and the question becomes how quickly you know. Traffic is a lagging indicator. By the time a drop shows in analytics the damage has been accumulating for days. The leading indicators sit earlier in the chain: crawl distribution in the server logs, the gap between sitemap URLs and indexed URLs, rendering parity on key templates, Core Web Vitals drifting on one page type. Monitoring those, with alerts weighted by the commercial value of what is affected, means you hear about a problem while it is still a technical one rather than a commercial one.

Reporting. The least glamorous of the four and the one most teams start with, which is a mistake if it is also where they stop. Automated reporting is worth doing when it removes the manual assembly of numbers so that the human time goes into interpretation. It is not worth doing when it produces a weekly deck that restates what the dashboard already showed. The test is whether the report changes a decision. If nobody acts differently because of it, automate something else.

I have built versions of this before

None of what follows is theoretical for me. Across my time at Iron Mountain and Staysure, and throughout the years running RichWeb Media, I built a number of these tools myself.

  • A pre publish checker for the content team. Before an editor could push a page live it was run against the SEO checklist automatically. Broken links, missing or oversized images, title and description length, empty alt tags. The editor saw a plain list of what to fix before the page went anywhere near the index.
  • A hreflang tool that could validate a single page or the whole site, checking that every declared alternate existed, returned the right language and pointed back. On a multilingual estate that is the check most worth having, because the errors are invisible to anyone reading one language.
  • Sitewide monitoring for the plumbing. Broken links, missing or incorrect canonicals, unexpected noindex, 404s and redirect chains, running continuously and reporting by section so that a regression showed up as a change rather than as a line in a quarterly audit.
  • At Staysure, an early detection layer that watched uptime, broken pages, 404s and movements in a defined set of rankings, both drops and climbs, and then wrote the report itself and sent it to the right stakeholders by email or message. The important part was the last step. A monitor that nobody is told about is a log file.
  • What I am building now, through my own research work, takes the same idea further. Content is loaded into embeddings and a graph so that semantically related topics surface automatically, and the most relevant internal linking candidates for any page are ranked and joined to GA4 and Search Console data. The result is a linking recommendation that knows which pages actually earn traffic and which ones the estate has quietly orphaned.

These are not unusual pieces of engineering. Each is a modest amount of Python, a scheduler, a datastore and a way of telling someone what was found. The pattern repeats because the underlying problems repeat, and once you have built one of these you can see where the next twenty belong. A classifier that spots the wrong language on a locale page. A script that validates structured data across every template nightly. A model that flags pages drifting toward duplication before a human would notice. A redirect map drafted from embedding similarity during a migration. The list is long, and almost none of it needs a budget line.

What the SEO team should hand to a machine

The wider SEO team is usually the group with the least engineering support and the most repetitive workload, so this is where automation pays back fastest. The principle is simple. Anything that is assembled rather than judged should be assembled by a script, and the output should arrive in a form a person can act on without reading a probability score.

The jobs I would move first:

  • Weekly health checks across the estate. Status codes, canonicals, meta robots, sitemap coverage and hreflang reciprocity, run on a schedule and reported by template and locale rather than by URL. Nobody wants a list of eight thousand pages. They want to know that the product template in the German site lost its canonical on Tuesday.
  • Crawl to crawl diffs. The interesting information is rarely in the crawl itself. It is in what changed since the last one. A diff that lists new URLs, lost URLs, changed titles and changed directives, grouped by section, turns a four hour review into a twenty minute one.
  • Content gap and coverage by market. On a multilingual estate the question of which locales are missing which content is a join between two lists, and joins are what databases are for. The judgement about which gaps matter stays with the team.
  • Prioritisation scoring. A backlog of technical issues is only useful if it is ordered. A simple model that combines severity, the commercial value of the affected URLs and the likely effort to fix produces a list the team can defend to engineering, and it does so consistently rather than according to whoever shouted loudest in the last meeting.
  • The assembly half of reporting. Pulling the numbers, building the tables, flagging the movements beyond normal variance. The commentary stays human, because the commentary is the part that carries judgement and the part the business actually reads.

What I would not automate at this level is anything that touches production. The SEO team should receive suggestions and draft tickets with the evidence attached, and a person should approve the change. The point is to free time for judgement, and judgement is not improved by removing the people who exercise it.

What the technical SEO team should build

The technical team owns the early warning layer, and the backbone of that layer is the server log. Everything else is a proxy. The log tells you what search engines actually requested, what they received and how that distribution shifted after a release, and it tells you days before Search Console does.

The systems I would expect a mature technical SEO function to run:

  • A log pipeline that classifies bot traffic by template, section and locale, and watches the distribution rather than the volume. A sudden rise in crawl of parameter URLs or a fall on a revenue section is actionable. A raw hit count is not.
  • Element change detection on a sample of URLs per template and per market, checking title, canonical, meta robots, hreflang, structured data and internal link counts on a schedule, with every change tied to the release that caused it. The release attribution is the whole value. An alert that says something changed is noise. An alert that says it changed in Thursday's deploy, on these three templates, and here is the diff, is a ticket.
  • Indexation drift. The gap between what the sitemap declares and what is indexed, sampled through the inspection API and tracked per section, so that a template quietly falling out of the index shows up as a trend rather than a surprise.
  • Rendering parity. The raw HTML against the rendered DOM for key templates, because this is the defect that hides longest and costs most.
  • Anomaly detection with seasonality built in. Fixed thresholds generate alerts every Monday and every bank holiday. A model that knows what a normal week looks like for each page group and each country only speaks when something is genuinely outside the expected range, and that is the difference between an alerting system the team trusts and one it mutes.
  • Hreflang generated from a single source of truth. On an estate in 35 languages the hand maintained hreflang matrix is not a system, it is a liability. Generate it from the content model, validate reciprocity automatically, and the entire category of error disappears.

The second responsibility of the technical team is to make all of this portable. Checks should live in configuration rather than in code written for one site, so that a new brand in the portfolio inherits the full set in a day. Every alert type should have a runbook that says what it means and who owns it. And the small set of things that must never ship anywhere, a template going noindex, a canonical pointing at another domain, a robots file blocking the lot, should be written down as guardrails that every pipeline enforces without discussion.

That last part is what turns a clever individual's toolkit into an organisational capability. A monitor only you can run is a hobby.

Developers need checks that run in minutes

Every engineering team already has a release pipeline with quality gates in it. Unit tests, integration tests, accessibility checks, a linter. If any of those fail the merge is blocked and the developer fixes it before anyone else is affected. SEO is almost never in that list, which is the single biggest reason large sites keep shipping defects that the SEO team then discovers a week later from a traffic chart.

The fix is to put SEO in the list, and the thing that makes or breaks it is speed. Developers will tolerate a check that runs in two minutes. They will route around one that runs in forty, however important you tell them it is. So the design has to be layered, with the fast checks on every pull request and the slower ones reserved for the staging build.

  • On every pull request, static checks with no crawling at all. Has robots.txt changed. Has the sitemap generation logic changed. Has anything touched the redirect rules, the hreflang configuration or the templates that set canonical and meta robots. Any of those flags the pull request for SEO review automatically. Add lint rules for the templates themselves, so that a canonical tag must be present and absolute, nothing sets noindex outside an agreed list, and structured data must parse. And check that nothing in the build references the staging domain, because that mistake has reached production on more sites than anyone admits.
  • On the staging build, the real gate. Render a fixed sample of URLs, one or two per template per locale, in a headless browser and compare each against the same URL on production. Status code, canonical, meta robots, hreflang set, title, headings, structured data types, internal link count and the parity between raw and rendered content. Every field carries a rule. Identical, allowed to change within a tolerance, or expected to change because the release says so. Hard breaks fail the build. A template going noindex, a canonical leaving the domain, a 200 becoming a 404. Soft changes warn and let the release through.
  • After the deploy, the same comparison against the live site within the first hour, plus a check in the logs that crawlers are still getting 200s on the affected templates. Staging rarely mirrors the CDN, the edge redirects or the middleware, so a clean staging run is not proof. Only production is production.

A few practical points decide whether this survives contact with an engineering team. The URL sample is the hard part and needs to be versioned alongside the code and regenerated when templates change. The output has to be readable by a developer who knows nothing about SEO, which means a table of field, expected, actual and URL, with no jargon. The gate should start in warning mode for a few weeks while the false positives are tuned, because a gate that blocks the first release it sees gets switched off within a month. And the ownership should be split cleanly. SEO owns the rules, in a config file the SEO team can edit, and engineering owns the pipeline step that runs them.

None of this is exotic. It is a modest script using a headless browser, a comparison library and a test report in the format the pipeline already understands. The engineering investment is days. The incident it prevents is the one that would otherwise have defined the quarter.

Where trained models help and where they do not

Most of what I have described is deterministic. A canonical is either present or it is not, and a rule catches it. Trained models earn their place in the parts of the work where the question is a judgement over a large volume of data, and there are a handful of those.

Classification at scale is the obvious one. On an estate of hundreds of templates and dozens of markets, a model that labels every URL by page type and intent, and flags where the language on a page does not match the locale it sits in, lets the monitoring group pages sensibly and lets a person look at twenty rows instead of twenty thousand. Near duplicate and thin content detection through embeddings belongs in the same category, run continuously rather than once a year.

Anomaly detection is the second. A model trained on your own history of what normal looks like, per page group and per country, with seasonality and the release calendar as inputs, produces far fewer false alarms than any fixed threshold and gets better as the history grows.

Triage is the third and the most interesting. When an alert fires, the diagnostic work is a playbook. Pull the affected URLs, fetch the rendered HTML, check the logs, check what deployed, check whether a known algorithm update landed, look at whether competitors moved. An agent can run that playbook in minutes and hand a person a ranked list of likely causes with the evidence attached. The person still decides. The agent has simply done the four hours of collection that used to stand between the alert and the decision.

Where I would keep models out is the gate itself. The decision to block a release should rest on a rule a developer can read and argue with, because a developer who is blocked by a probability score will not trust the system and will find a way past it. Models can decide how deep the checks run for a given change, by scoring its risk against what previous changes of that shape did to traffic. They should not be the thing that says yes or no.

The other discipline is honesty about false positives. Every model will be wrong some of the time, and the team needs to know roughly how often and in which direction. Report that figure alongside the model's output. A tool whose error rate is known is a tool people can use. One whose error rate is hidden is one they will eventually stop believing.

The measure of the system

If I had to pick one number to judge all of this by, it would be the time between a defect being introduced and the right person knowing about it. On most large estates today that gap is measured in weeks, and the cost of those weeks is paid in traffic and in the credibility of the SEO team with the people who run engineering. Every tool described here exists to shorten it. The prevention layer takes the gap to zero for the defects it covers. The early warning layer takes it from weeks to hours for the rest. The reporting and triage layers make sure that what arrives in those hours is something a person can act on.

None of it replaces the judgement of experienced people. It gives that judgement something better than a quarterly photograph to work with.

Related on this site: Technical SEO at Scale Needs More Than Another Audit, SEO Test and Learn: Building Evidence About Your Own Site and Keeping Staff Out of Your Analytics When IP Filters No Longer Work.

Sources