How to measure whether content changed AI answers

Guide · Continuous Intelligence · 5 min read · last verified 2026-07-27

To measure whether a piece of content changed AI answers, you need three things in place: a question set that was locked before the content went live, a recorded baseline of what the answers said at that moment, and a re-scan of the identical questions after publication — with competitor movement tracked alongside as a control. Anything less produces a story, not a measurement. The method is simple to describe and easy to get wrong, because AI answers move on their own for reasons that have nothing to do with your publishing, and an unstructured before-and-after glance will happily credit your blog post for a model update.

Why "did my content work?" is harder than it looks

When an AI answer changes in your favour the week after you publish, there are at least four candidate explanations: your content was retrieved and used, the underlying model was updated, the pool of sources the answer draws on churned, or a competitor's page dropped out of retrieval. Only one of those is your doing. The other three operate continuously and independently — assistants ship model revisions on their own schedules, third-party pages get published and re-ranked daily, and rivals are running their own content programmes.

This is the attribution problem in miniature. Classic marketing attribution struggles because many touches precede one purchase; AI answer attribution struggles because many forces precede one changed answer. The response is the same in both cases: you cannot eliminate the confounds, but you can design the measurement so they are visible instead of invisible.

Lock the question set before the content ships

The single most important discipline is choosing your measurement questions before you see any results. If you scan broadly after publishing and then pick the questions that improved, you are selecting winners from noise — answers vary between runs, and a wide enough net will always catch a few favourable changes.

A locked set works like pre-registration in research. Before the content goes live, write down the exact buyer questions the piece is meant to influence, plus a handful it is not meant to influence. Record the baseline for all of them: which companies each answer names, how your company is described, and which sources are cited. That frozen snapshot is the only honest reference point. Magrios builds this in — a benchmark locks the question set and phrasing at baseline, and every later scan re-runs the identical set, so the comparison is never quietly redrawn around a good result.

The before-and-after protocol

The working sequence is short:

• Baseline. Scan the locked set and record presence, description, and citations per question.
• Publish. Ship the content, and note the date against the benchmark.
• Wait a realistic interval. Assistants need to crawl, index, and begin retrieving a new page. In our observation this is measured in weeks, not days, and it varies by site authority and by how each assistant sources answers. Watching daily mostly measures noise.
• Re-scan the identical set. Same questions, same phrasing.
• Compare per question, not in aggregate. A single averaged score can hide the one question that actually moved.

For each target question, record four things: whether you now appear where you did not, whether the description of your company changed, whether your new page shows up among the citations, and whether the answer's wording echoes phrasing that only your content uses. Those four observations carry very different evidential weight, which is the subject of the next two sections.

Competitor movement is your control group

The untargeted questions in your locked set, and the competitors tracked across all of it, are what let you read the re-scan honestly.

The logic is a natural experiment. Your content targeted specific questions. If those improved while the untargeted questions and competitor positions held roughly steady, the change is plausibly yours. If everything moved at once — your presence, rivals' presence, questions your content never touched — the likely cause is a model update or broad source churn, and claiming credit would be self-deception. If only one question moved but the citation list shows a third-party roundup appearing or vanishing, source churn is the better explanation.

Broad simultaneous movement points away from you; narrow movement on exactly the questions you targeted, with your URL in the citations, points toward you. Without a control, both patterns look identical: "the answer changed."

What counts as strong evidence

There is a rough hierarchy of evidence that your content caused the change, from strongest to weakest:

Be honest about the ceiling: this is observational measurement, not a controlled experiment. You cannot randomise which buyers ask which assistant, and a single re-scan is a single sample of a system with run-to-run variance. Two practices raise confidence: re-scan more than once before concluding anything, and treat repeated patterns across scans as the unit of evidence rather than any single answer. When we say a piece of content "moved" an answer, we mean the movement persisted and the citation trail supports it — not that causation was proven.

How long before content shows up

There is no fixed timeline, and any vendor quoting one is guessing. The lag depends on how quickly your page is crawled, whether the assistant retrieves live or leans on periodic indexes, and how contested the question is — a question with few good sources admits new evidence faster than one with an entrenched set of citations. The practical answer is to re-scan on a steady cadence rather than to watch for an arrival date, and to keep the benchmark open long enough that a slow retrieval pipeline does not get scored as a failure. How often to re-run the measurement is its own question, covered in the re-scan cadence piece linked below.

When nothing changed

A flat re-scan is information, not just disappointment. Work through the explanations in order. The page may not be retrieved yet — check whether it is indexed and whether any assistant cites it on any question. The question may be heavily contested, with strong incumbent sources your single page cannot displace. Or the content may not answer the question in a liftable form — engines quote passages that answer directly, and a page that circles its subject gives them nothing to take.

Each diagnosis has a different response: wait and re-scan, add supporting evidence on other surfaces, or refresh the page so the answer is quotable. What a flat result never justifies is abandoning the measurement — the piece that eventually moves the answer will need the same locked baseline to prove it. For the business-level layer above per-piece measurement, and for the phenomenon of answer drift itself, see the linked articles on AEO ROI and on how often AI answers change.