Why does marketing data always look better than reality?

Beyond the three filters described above, there is a structural incentive common to all three: the person assembling a metric and the person whose performance it will be used to judge can be the same person, and that dual role shapes small, defensible choices — which period to compare against, which segment to include — long before any number is deliberately altered. None of those choices need to be dishonest individually to add up to a number that flatters its author.

What biases distort marketing measurement?

Alongside the three above, watch for denominator neglect: a rate or count reported without the base it was computed over. A count of new signups from a campaign sounds impressive on its own, but means something very different depending on whether it came from a small, targeted send or a massive one, and the standalone count carries the same confident tone either way. That is the reporting-side twin of the display argument in Market dashboards that lie less; the rule either way is to treat a rate or count presented without its denominator as incomplete rather than merely imprecise.

How do I correct for survivorship bias in case studies?

Correcting for it does not require rejecting the case study, only pairing it with something the write-up will not volunteer on its own: ask what criteria selected this particular customer over every other one the team could have picked, and whether a similar account that did not produce this outcome is missing from the shortlist entirely. A team that chose the example because it was the strongest available result should be able to say so directly; a team that cannot describe how the example got picked is usually presenting a coincidence as a pattern.

Why marketing data flatters itself

Guide · Frameworks · 6 min read · last verified 2026-07-29

Reviewed before publication Editorial board Independent commercial review

In shortMarketing data looks better than reality for structural reasons, not dishonest ones: the data that survives to be reported has already passed through a filter that favors good news, invisibly.

Marketing data looks better than reality for structural reasons, not dishonest ones: the data that survives to be reported has already passed through a filter that favors good news, invisibly. Three of those filters are specific enough to name individually — which examples get written up, who bothers to answer a survey, and channels measuring their own contribution — each a mechanism that pushes one way, not a vague complaint about spin. This is about distortions in the numbers themselves, a level below the skepticism applied to outside claims in How to evaluate marketing advice — that piece questions claims from outside a company; this one questions data a team already trusts because it is their own.

Survivorship: the wins that survived to tell you about it

A case study is not a random sample of what happened; it is a sample of what worked, written up by a team with a reason to write it up. For every published success story, an unknown number of customers tried the same product, in similar conditions, and got a result no one wrote up — not because the result was hidden, but because a middling outcome gives a customer no reason to agree to be quoted. The distortion is not in any individual case study, which may be entirely accurate; it is in treating the visible cases as representative of the entire population that could have produced one.

The corrective question is simple to ask, and it is not the question a write-up is built to answer: how many customers matched this profile, and how many got a result like the one being shown? A single glowing example says a good outcome is possible. Without a denominator attached it says nothing about how often that happens, and a document assembled to make a case is not built to carry one.

The same filter runs through win-loss research, where every interview available comes from a buyer who entered your process in the first place. Win-loss analysis vs market intelligence works that population through and says what seeing the rest of it would take.

Response bias: survey answerers are not customers

A survey response rate below completeness means the respondents are a self-selected group, not a random one, and self-selection correlates with the exact things a survey wants to measure. Satisfied customers reply to prove a point; frustrated customers reply to vent; the large, silent middle — neither thrilled nor angry enough to spend five minutes on a form — mostly does not reply at all. The average of the responses received describes the respondents, not the customer base, and that gap is precisely the information a low response rate throws away. Why NPS does not predict renewal is a worked example of this mechanism applied to one metric in particular.

This is not an argument against surveys; it is an argument for treating a response rate as part of the result, not a footnote beneath it. A satisfaction score attached to its response rate is more useful than the same score reported alone, because the reader can weigh how much of the silent middle the number actually describes. Surveys vs interviews for customer research reaches the same limit from the method side: a form that records only the people willing to fill it in answers a smaller question than the one it gets asked, which is why a small set of interviews and a large survey are not substitutes for each other.

Grading its own homework

An ad platform reports how much credit it deserves for a result, and the incentive built into that arrangement is not subtle: a platform that reports itself as less effective loses budget to one that reports itself as more effective, whether or not the underlying difference is real. The reported number is not fabricated in the ordinary sense — it reflects the platform's own crediting rules applied consistently — but those rules have an author with a stake in the outcome, and that is the part worth holding onto.

The tell is positional. When a channel sits close to the moment of purchase, demand that earlier and harder-to-measure work built up arrives already formed, and the last recorded touch takes the credit for it — the platform is not lying about what it saw, only about what that sight means. Whether a credited result would have arrived without the channel is the question incrementality is built around, and there are test designs for answering it. What this section adds to that piece is the conflict of interest alone: the rules that decide what counts as a contribution were written by the party whose budget depends on the answer.

The question that catches all three

All three mechanisms share a single blind spot, so one habit reaches all three: before trusting a number, ask what did not make it into this number. For a case study, that is the customers who did not get written up. For a survey, it is the people who did not respond. For a platform's self-reported result, it is the activity elsewhere the platform has no way to see and therefore no way to credit.

The question works because each distortion operates the same way — not by falsifying what is shown, but by controlling what gets shown in the first place. A number that has passed through a filter can be completely accurate about the cases it covers and still misrepresent the wider population, because the filter already removed everyone who would have told a different story.

What this piece is not

This is a piece about distortions that creep into measurement generally, not a how-to for any single technique. Cohort analysis and holdout tests are both legitimate methods for answering specific questions cleanly, but both are still built from inputs that can carry the same three distortions above. A cohort table built from a segment that excludes customers who left before the data was pulled has survivorship baked into its foundation before a single row is drawn; a holdout comparison contaminated by a self-selected leak has response bias by another name. The techniques are not the disease and not the cure — they are structures that still need the corrective question applied to whatever feeds them.

Applying the same question to a visibility score

The same question is worth turning on any tool that claims to measure something, including this one. A visibility scan like Magrios reports which sources AI assistants currently cite across a defined, repeatable list of buyer questions, and that number is only as meaningful as the sample behind it — a citation figure is readable only beside the question set it came from and the date it was scanned. Strip those two away and it takes the shape of a case study with its denominator removed. It can be accurate about everything it counted and still imply more than the counting will support.

Once a number survives these filters, displaying it honestly is a separate problem: Market dashboards that lie less is the display-side version, where the fix is to keep the base, the method, and the observation date beside the figure. No dashboard design, though, repairs a number distorted before it ever reached the screen.