When should we update our benchmark questions?

Only when a trigger you can demonstrate outside your own dashboard fires: buyers stopped asking in the old vocabulary, your product scope genuinely changed, or a question died structurally and no longer distinguishes anyone. A disappointing score, stakeholder preference, novelty, or the passage of time alone are not triggers — they are how benchmarks drift into meaninglessness.

Does the market outgrow a locked question set?

Eventually, yes — markets rename themselves and product scopes shift — but usually more slowly than teams itch to change it. A set built from durable, evidenced buyer questions can stay valid for a long time. The signal that the market has genuinely moved is external: sales-call language, community discussions, and the words prospects bring, not anything on the dashboard itself.

How do we change questions without breaking the trend?

You don't avoid the break; you mark it. Keep the old set running, draft the new set with the same evidence discipline, run both in parallel across several scan cycles so the overlap calibrates the difference, then publish the new trend as a new version with the break visible in every chart. Splicing the versions into one continuous line misrepresents the measurement.

Can we just add a few questions instead of replacing the set?

Additions are changes too. New questions move the score by changing what is being averaged, without your market position moving at all, so an expanded set is a new version and needs the same treatment: a marked break and a parallel read against the old set. Quietly growing the set is one of the more common ways a trend stops meaning anything.

When to change a locked question set

Guide · Continuous Intelligence · 5 min read · last verified 2026-07-27

Reviewed before publication Editorial board Independent commercial review

In shortThe lock is what makes a visibility trend mean anything, so question-set changes are versioned, never silent: real triggers, the parallel-run procedure, and how to read a trend across a marked break.

A locked question set is the fixed list of buyer questions a visibility benchmark re-runs on every scan, held constant so that a change in results means the market moved rather than the measurement. Sooner or later the market moves underneath the lock — a category renames itself, a product line ships or dies — and the set has to change. The doctrine for that moment fits in one line: changes are versioned, never silent. This page covers what the lock buys, which reasons for changing it are legitimate, and the procedure that changes the set without corrupting the record.

What the lock buys — and what a change spends

A trend line means something only when both of its endpoints were measured identically; that is the entire argument of why trend lines need fixed methodology, and the locked set is how a visibility benchmark honors it. Every change to the set spends trend history: comparisons that cross the change are no longer clean, and whatever story the old line was telling ends there.

So the bar for changing is not "would new questions be better" — newer questions are almost always arguable improvements — but "does the added relevance buy more than the destroyed history costs." Stated plainly, because it is the operating principle behind everything below: trend integrity outranks the score. A benchmark that keeps its questions honest and its history intact is worth more than one that reads better this quarter.

The wrong reasons to change it

The score is disappointing. Editing questions after a weak scan is choosing the measurement to fit the desired result, and everyone who later learns the timing will discount the trend — correctly. The adjacent temptation is quieter: expanding the set until the number looks healthier. Adding prompts changes your score without changing your position covers why that move fools only the people running it.

A stakeholder prefers different phrasing. Preference is not evidence that buyer language changed. The set exists to mirror how buyers actually ask, which was established with evidence when the set was built — the discipline described in what is a benchmark question set — and it takes evidence, not taste, to revise.

Something new appeared. A new tool, a new report, a competitor's framing — novelty is not obsolescence. Durable buyer questions outlive most of what gets said about them.

Time passed. Age alone is not a trigger. A set built from real pre-purchase questions can stay valid for a long time, and "it has been a while" is how sets drift into churn.

The legitimate triggers

Each real trigger shares one property: you can demonstrate it happened outside your own dashboard.

The market renamed itself. Buyers stopped asking in the old vocabulary — visible in sales calls, community discussions, analyst language, and the words prospects bring to first meetings. When the set's phrasing measures a conversation nobody is having anymore, the lock has outlived the language.

Your scope changed. A product line shipped, or one was retired. The set now covers offerings you no longer sell, or is silent on ones you do. The benchmark should measure the business you are, not the one you were when it was built.

A question died structurally. A regulation redrew a category, two markets merged, a question stopped distinguishing anyone because every credible vendor now clears it. A question that can no longer separate outcomes is ballast, not measurement.

The versioned-change procedure

When a real trigger fires, the change is an event in the record, not an edit to it.

Keep the old set running. Do not stop it the day the decision is made; its continuity is about to become useful. Draft the new set with the same evidence discipline the original demanded — real buyer questions, phrased as buyers phrase them, with sources noted. Then run both sets in parallel across several scan cycles. The overlap is the calibration: it shows how much of the difference between old and new numbers is the set itself, and whether the two move together when the market moves.

Publish the new trend as a new version, and mark the break in every chart that shows history — visibly, so no reader can mistake the two lines for one. A chart drawn continuous across a set change is a quiet claim that both sides were measured identically, and that claim is false. Retire the old set only after the overlap has done its calibration work, and archive its full history; deleted history is the other form of silent change. This is how Magrios handles set changes: versioned runs, a marked break, and the old line kept.

Reading a trend across a version break

Within a version, movement is attributable — that is what the lock is for. Across the break, claims must shrink to what the parallel run supports. If old and new sets moved together during the overlap, directional statements that span the break carry reasonable weight; if they diverged, the honest reading is that the two versions measure differently, and cross-break comparison stops at "different instruments." What is never legitimate is splicing the versions into a single line and letting the reader assume continuity. The break is information; hiding it converts a methodology change into a lie about the market.

Who may change it, and how often

Give the set a named owner, and require changes to be proposed in writing with the outside evidence attached — the sales-call language, the community discussions, the shipped product. Rarity is a feature: if the set wants changing every few scans, it was built on fashionable phrasing rather than durable buyer questions, and the fix is a better-built set, not faster churn. One timing rule earns special mention: infrastructure upheaval is the worst moment to also change the measurement. Through a replatform or domain move, hold the set absolutely still — how site migrations affect AI visibility depends on a before-and-after comparison that only an unchanged set can provide.