The technical audit lands as a spreadsheet. Forty thousand rows, colour-coded by severity, a summary tab that counts the errors, and a covering note that says the site has significant technical debt. Everyone in the room already knew that. What nobody in the room knows is which of the forty thousand rows cost the business anything, which were introduced by the release that went out on Tuesday, which will be reintroduced by the release going out next Tuesday, and which are the same defect reported four thousand times because it lives in a template. The audit is accurate and almost useless, and in a few weeks it will also be out of date.

I have commissioned, written and received a great many of these documents, and I still think they have a place. A good audit gives a team visibility of crawling, indexation, rendering, canonicalisation, international architecture and performance that it would not otherwise have. But it describes a site on a particular day, and across a portfolio of sixty or more sites on several platforms with several engineering teams, the day is over before the findings are read. The difficult part of enterprise technical SEO has never been finding another problem. It is knowing which problems matter, responding before they compound, and building something so the same investigation does not have to be repeated by hand every quarter. Python, open-source tooling and smaller AI models have changed what is possible there, and I want to set out how, without pretending that the tooling does the thinking.

On this page

  1. Complexity grows with the estate
  2. Commercial judgement decides which problems matter
  3. Continuous technical intelligence with Python
  4. Smaller specialist models and where AI belongs
  5. What building DMCI has taught me
  6. From automated findings to reliable decisions
  7. Building SEO into Product and Engineering
  8. Test, learn and improve the system itself
  9. Capability that scales
  10. References

Complexity grows with the estate

On a single site, a technical problem is usually a technical problem. On a portfolio it is a pattern, and the pattern is the thing you need to see. Two examples from the kind of estate I have run will show why isolated inspection fails.

The first is canonicalisation. An HTML crawler fetches a page, finds a self-referencing canonical in the source and reports it as correct. The same page, rendered, executes a script that rewrites the canonical to a parameterised URL, or to the English version, or to nothing. Google's own documentation is explicit that it processes JavaScript in a second rendering stage and that what it indexes is the rendered result, so the crawler's green tick describes something Google never sees. On one template this is an oddity. Across a shared template used by thirty markets it is thirty markets quietly consolidating into the wrong URLs, and the symptom in Search Console is a slow drift in indexed pages that nobody connects to the release that introduced it.

The second is hreflang. Every individual URL can carry annotations that look correct when inspected on its own. The relationships between them are where the errors live: a cluster where the German page points to the French page but the French page does not point back, a canonical that points outside the cluster and breaks it, a market that was added to the site without being added to the annotation set. These are only visible when the whole cluster is assembled and checked as a graph, which no page-level report does, and on an estate with 35 languages the number of relationships to check runs into the millions. The same shape applies to internal linking, redirect chains after migrations, duplicate content between brand and country domains, and performance regressions that only appear on one CMS. The lesson I took from years of this is that an apparently trivial implementation difference, repeated across templates and markets, is the typical enterprise technical SEO problem, and that meaningful analysis means inspecting relationships and rendered states rather than accepting what a page-level crawler reports.

Commercial judgement decides which problems matter

An enterprise crawl will produce thousands of warnings, and a tool that reports a warning has not established that there is a problem. The most important technical SEO skill at scale is deciding which findings deserve engineering time, and it is a commercial skill as much as a technical one. A missing meta description on eight thousand archive pages that nobody searches for is noise. A canonical pointing at the wrong URL on forty pages that generate a third of the organic enquiries is an incident. The crawler ranks them the other way round, because it counts rows.

Question What it changes
Commercial importance Do the affected pages produce revenue, leads or acquisition, or are they inventory?
Scale How many commercially relevant URLs, templates or markets carry the problem?
Severity Does it stop content being discovered or indexed, degrade it, or just look untidy?
Confidence Is this verified in the rendered page and in Search Console, or is it a single crawler's opinion?
Effort What does the fix cost in engineering time, and does it need one team or four?
Opportunity cost What does not get built while this is being fixed?

The distinction I keep coming back to is between hygiene and opportunity. Hygiene is work that should be done to an agreed standard whether or not anyone can attribute traffic to it, because leaving it creates operational risk: broken hreflang on an international estate, a robots directive that could block a section, a redirect chain that will fail under the next migration. Opportunity is work where there is a reasonable expectation of a commercial result and that expectation should be tested. The two are prioritised differently and reported differently, and most audits do not tell you which is which.

Automation belongs in this decision as a supplier of evidence, not as the decision maker. A system can tell you how many pages, which templates, which markets, what the pages earn and whether the finding is confirmed in the rendered state. Whether that adds up to the next sprint's top item is a judgement, and I have not seen a tool that makes it well.

Continuous technical intelligence with Python

I have written Python for most of my working life and the thing it is best at in this context is unglamorous: collecting the same information from several sources on a schedule, and telling you what changed. The sources are usually already available. A crawler's export, a headless browser rendering a sample of templates, Search Console through its API or the bulk export into BigQuery that Google has offered since 2023, the server logs that show what Googlebot actually requested and how often, structured data validated against the schema each page type is supposed to carry, the CMS's own API, and the performance data from field and lab tools. None of these is hard to reach. What is rare is a system that reads all of them for the same URLs, keeps the history, and compares today with the last known good state. Logs deserve a particular mention because they are the only source that reports behaviour rather than markup: a template can be perfectly formed and still not be crawled, and nothing but the logs will tell you.

The checks worth running continuously are the ones that catch regressions a release can introduce in an afternoon.

  • Canonical URLs changing on a template, in the rendered page as well as the source.
  • Noindex or disallow directives appearing on pages that earn money.
  • Important internal links disappearing from navigation, footers or related-content blocks.
  • Content present in the source but absent from the rendered HTML.
  • Hreflang clusters losing reciprocity, or a market dropping out of the set.
  • Redirect behaviour changing, including chains and loops after a migration.
  • Core Web Vitals field data deteriorating on a template after a deployment.

The design principle that makes this useful rather than noisy is that the system compares against expectation, not just against yesterday. A canonical that changed because a product was consolidated is expected; the same change on a page nobody touched is a regression. Weekly variation in crawl frequency is routine; a 40% fall on one template after a release is not. Getting that distinction right is most of the engineering effort, and it is the difference between a monitor people trust and an alert channel they mute within a month.

Once a change is detected, the same code can do the first hour of investigation a specialist would otherwise do by hand: pull the affected URLs, group them by template and market, fetch the rendered state, compare with the historical values, attach the Search Console and log figures, and write the finding up with its evidence in the format the engineering team's tracker expects. That turns monitoring into an early-warning system, in the sense that a material issue is known, scoped and costed before it has had time to become a commercial problem. The most durable version runs the same checks as quality assurance in the release pipeline, so the regression is caught in staging on Tuesday rather than in next month's traffic. All of this is deterministic. It checks facts, compares values and calculates differences, and it should stay that way, because it is the foundation anything cleverer has to stand on.

Smaller specialist models and where AI belongs

The assumption that enterprise automation needs the largest available model for every task has cost organisations a great deal of money and not much accuracy. Most of the analytical problems in technical SEO are narrow. Classify this page as product, category, support or editorial. Group these four hundred crawler findings into the six underlying defects. Decide whether this change in page content is a rewrite or a template refactor. Summarise the evidence gathered on this cluster into two paragraphs an engineer will read. None of these requires a frontier model. Each requires a model that has been chosen, adapted and evaluated for that task, and a smaller model treated that way will usually outperform a larger one that has been handed a prompt and trusted.

Open-weight models have made this practical. Serving software such as vLLM runs them on company infrastructure with structured outputs that slot straight into a Python pipeline, so the model returns a category or a JSON record rather than prose that has to be parsed. The case for self-hosting is control: over where the data goes, which model is used, what the processing costs, and how the model is integrated with internal systems. The case against is that self-hosting is an operational commitment. It needs infrastructure, monitoring, security review, evaluation and someone who owns it when it breaks, and it is not automatically cheaper or more accurate than an API. Whether it is the right choice depends on company policy and capability, and I have seen it go both ways.

Over the past three years I have trained more than fifty specialist models, which has shaped how I think about this. The attraction of a task-specific model is not that it is clever. It is that it can be measured. You define the task narrowly enough that the categories are unambiguous, you build an evaluation set that represents the real inputs, and you find out whether the model is right often enough to be useful before it touches a decision. A general-purpose model used without that discipline will give you a plausible answer every time and no way of knowing which answers to trust.

So the division of labour is simple to state. Python and deterministic code check facts, compare values, calculate changes and enforce rules. Models do classification, language understanding and semantic interpretation, where there is no rule to write. The two complement each other, and the engineering choices matter far more than the model sizes.

What building DMCI has taught me

Much of my recent time has gone into DMCI, a personal research and development project on enterprise content and market intelligence. It is not a technical SEO auditing tool and it is not a finished product. It is an agentic system, still being built, that explores how specialist models, governed evidence, retrieval, deterministic processing and verification can be combined to answer complex questions about large content estates and markets. I mention it because building it has taught me more about what works in applied AI than any amount of reading, and most of the lessons transfer directly to SEO automation.

The problem it started from is one I described in an earlier article: large organisations hold enormous amounts of information across sites, languages, services and markets, and understanding it consistently is hard. Every investigation is done by hand, by someone who knows where to look, and the knowledge leaves when they do. DMCI asks whether a system can gather the evidence, classify it and answer the question, with every step traceable back to what was found.

The most instructive work has been training and evaluating models to recognise information according to defined classifications. The difficulty is never getting a model to return a category with a confident explanation. It is establishing whether the category is correct, whether the model had enough evidence to reach it, and what the system should do when it did not. That has meant long stretches on classification definitions, because a model cannot be right about a category the humans have not agreed; on training examples that reflect the awkward real cases rather than the clean ones; on evaluation sets that are kept apart from training and that get harder as the model improves; and on deciding, with a measured error rate in front of me, whether more training is justified or the task itself was badly framed. More than once the answer was the task.

The agentic layer adds a different kind of difficulty. Connecting a language model to a set of tools is an afternoon's work. Organising an investigation so that the system knows which evidence is relevant, reconciles sources that disagree, represents its own uncertainty and refuses to present an assumption as a finding is the actual engineering, and it is where most of the design effort has gone. The principle I settled on early and have not had reason to abandon is that models are not allowed to establish numerical or technical facts that software can calculate and verify directly. A model can interpret, summarise and propose. The evidence store and the validation code are authoritative, and if the two disagree, the model loses.

Three conclusions follow for technical SEO. Specialised models can take a real share of the repetitive interpretation off specialists: grouping findings, classifying pages, drafting the write-up. Their outputs must stay attached to the evidence that produced them and be independently checked, or the saving is illusory. And the more automated the pipeline, the more engineering discipline it needs, because a wrong automated conclusion about a template is applied to every page on it. The technology is increasingly accessible. Building something reliable enough for an enterprise to act on is where the work begins.

From automated findings to reliable decisions

The failure mode of SEO automation is a system that goes straight from a crawler observation to a recommendation, and a team that acts on it. An observation, a diagnosis and a decision are three different things with three different confidence requirements, and the system should keep them apart.

Stage What happens Who or what does it Confidence needed
Observation A scheduled crawl finds canonical URLs changed on a group of pages Deterministic code That the data is correct
Investigation Affected pages, template relationships, rendered state, history and Search Console figures are gathered Code, with a model grouping and summarising That the evidence is complete and attached
Diagnosis The change is expected, a defect, or needs a human look Model proposes, rules and thresholds check Enough to route, not enough to act
Decision Business impact is weighed and the next step chosen A specialist Enough to spend engineering time

A few practices make this hold up in practice. Every finding carries its evidence and its source, so a specialist can see what the system saw rather than taking its word. Model-generated classifications are sampled and checked, and the error rate is known. Conflicting or uncertain results are surfaced as such rather than resolved by the most confident-sounding option. Alerts are rare, because every unnecessary one trains people to ignore the next. Changes and decisions are recorded, so when the same pattern appears on another site there is a precedent. And specialists can overrule the system and say why, which is both a safety valve and the best source of training data there is. The system also has to be maintainable and secure in the ordinary engineering sense, since it reads from and writes to production tooling; the OWASP guidance on AI-enabled applications is a sensible checklist for the model-facing parts. None of that needs to be elaborate. It needs to be present.

Building SEO into Product and Engineering

The best diagnostics in the world are worth nothing if the findings sit in a queue. And the most valuable fix is rarely the one that corrects the pages; it is the one that corrects the system that produced them. A recurring canonical defect that lives in shared template logic can be patched on the forty pages where it was noticed, and it will be back on the next deployment. Fixing the template removes it from every page on every market and from every future release. Most of the technical SEO improvement I have seen hold over time came from this kind of change, and it is why the conversation has to move from the SEO team's backlog into Product and Engineering's.

In practice that means a short list of things agreed with engineering leadership rather than requested ticket by ticket.

  • SEO requirements written as acceptance criteria, in the engineering team's own tracker and format.
  • Automated checks in the development workflow, so a canonical, directive or hreflang regression fails in staging.
  • Technical validation as a gate before major releases and migrations, with a named SEO reviewer.
  • Shared standards for templates and architecture that new work is built to.
  • Monitoring after deployment, with the results going back to the team that shipped the change.
  • Fixes built once as reusable components and applied across sites, rather than solved separately per platform.

The part that is not technical is influence. SEO specialists rarely control engineering capacity, and they do not win it by being right about canonicals. They win it by explaining a technical risk in terms an engineering director can weigh against the product roadmap, by understanding what else the team is being asked to build, by challenging a decision constructively when the evidence supports it, and by bringing enough of that evidence that the decision can be made quickly. At this level you lead through expertise and the quality of your case, because formal authority over the people who ship the code is not on offer and would not help if it were. The aim is for technical SEO to become a normal consideration in how software is built, with the SEO function in the room for architecture decisions, migrations and experiments from the start.

Raising the capability of the wider team

The tools described here only pay back if people other than their author use them. A monitor that one specialist understands is a single point of failure with a dashboard. So the work includes the unglamorous half: documenting what each check looks for and why, writing the diagnostic scripts so a country SEO lead can run them against their own site without reading the code, keeping a library of worked investigations that show how a finding was traced to a template and what was done about it, and spending time coaching specialists through their first few uses until the questions stop. In the operating model I have written about elsewhere, this is where the stand-ups and retrospectives earn their keep, because they are where you discover which tool nobody understands. The measure of a principal-level technical specialist is not how much they can do alone. It is how much the organisation can do that it could not before they built the thing.

Shared standards across a portfolio

Across many sites on different platforms, identical implementations are neither possible nor desirable. What can be consistent is the expected behaviour: how canonicals resolve, how international pages reference each other, what a redirect chain may never do, what structured data a page type carries, what is and is not crawlable. Standards written at that level let each engineering team choose an implementation that suits its stack while the SEO function tests every site against the same expectations. The supporting assets are the same ones that make the monitoring work: reusable specifications, a validation library, diagnostic scripts, and implementation patterns that proved out on one site and were documented for the next. On an estate of sixty sites, a solution found once and reused is worth more than a brilliant fix applied once.

Test, learn and improve the system itself

A monitoring system and a set of standards are themselves hypotheses about what matters, and they should be treated the way I have argued elsewhere that SEO changes should be treated. For an improvement with a plausible commercial upside, test it on a limited group of pages, read the evidence against a comparison, and expand when the result has repeated. For a defect that creates operational risk, correct it safely and validate it, and record it as hygiene rather than inventing an uplift. The two are different kinds of work and the system should know which it is handling.

The same discipline applies to the automation. If a model keeps misclassifying a particular kind of finding, that is a test result: investigate why, look at the examples it was evaluated on, decide whether the definitions were wrong, the examples were thin, or the task needs splitting, and only then retrain. If the monitor keeps raising alerts that specialists dismiss, the thresholds or the expectation model are wrong and the dismissals are the data that says so. Every investigation, release and migration teaches something about how the estate behaves, and the point of building a system is that the lesson goes into the system rather than staying in the head of whoever happened to be on call.

Capability that scales

Audits will keep arriving, and the good ones will still be worth reading. But a document that describes a site on one day is a small part of managing a large estate, and organisations now have a realistic option to build something better internally: a capability that watches the sites continuously, surfaces the material issues with their evidence, does the first hour of investigation automatically, and feeds what it learns back into standards and release checks. Experienced technical SEO people can build a good deal of this themselves, with Python, open-source serving and models small enough to evaluate properly, working alongside the engineers who own the platforms. The return is not mainly time saved. It is visibility across the whole portfolio, fewer recurring problems, and improvements that can be applied to sixty sites because they were built once.

I still value the detailed technical investigation that has always been part of this job, and I would not want to lead a team that could not do it by hand when it had to. What interests me now is how much of that knowledge can be built into the systems we run, so that the people who have it spend their time on the problems that need them.

Related on this site: Why Enterprise Content Teams Struggle to Understand Their Own Websites, SEO Test and Learn: Building Evidence About Your Own Site and The Global SEO Agile Playbook.

References