There is a number on your AI visibility dashboard that says something like "43% brand visibility." It looks precise. It is not.
That number is not a measurement of what AI tools told your buyers. It is an estimate, produced by simulation, and the gap between those two things is the most important thing to understand before you act on any AEO report, hand one to a client, or take one into a board meeting.
This is not an argument against the tools. Some of them are very good, and the category is now serious: Profound, one of the larger players, raised $96 million at a billion-dollar valuation in early 2026. The point is narrower and more useful. If you understand how these tools generate their numbers, you can tell a trustworthy figure from a confident-looking but hollow one, and you can ask the right questions before you spend money or stake a decision on the output.
The thing nobody puts on the sales page
Every AI visibility tool faces the same hard limit: it cannot see what real people ask.
When a buyer opens ChatGPT and asks which firms they should consider in your category, that conversation happens inside their private account. It is invisible to the AI platforms' own analytics and to every third-party tool on the market. There is no equivalent of search query data here. Nobody gets to watch the real questions.
So the tools do the only thing they can. They simulate. They build a list of prompts a buyer might plausibly ask, run those prompts against the models on a schedule, and read the answers to see whether your brand shows up, where, in what light, and which sources got cited. Everything on the dashboard is built from that simulation. It is a reasonable approach. It is also a model of reality, not reality itself, and it should be read that way.
The evidence that it's an estimate
You do not have to take this on faith. SparkToro ran one of the more rigorous public studies on AI answer consistency: 600 volunteers ran 12 prompts through ChatGPT, Claude, and Google's AI a combined 2,961 times in late 2025.
The result is striking. There was less than a 1 in 100 chance that any of these tools returned the same list of recommended brands on two runs of the same prompt. Getting the same brands in the same order was closer to 1 in 1,000.
Sit with what that means. The same question, asked twice, usually produces two different answers. So the old SEO instinct, "where do we rank," simply does not transfer. You are not in position three for a given query. You appear in some share of responses to it, and that share moves on its own even when nothing about your content or your competitors changes.
This is why a single dashboard figure is the wrong unit. "Our mention rate is 43.7%" tells you very little, because you have no stable baseline for what 43.7% means in absolute terms. "Our mention rate on high-intent prompts rose twelve points this quarter" tells you something real. The trend is the signal. The decimal point is decoration.
The four things that decide whether a score means anything
Two tools can report very different numbers for the same brand on the same day, and neither is necessarily lying. The difference usually comes down to four design choices. These are the questions worth asking about any tool, including your own.
1. Where do the prompts come from? This matters more than anything else, because a visibility score is only as meaningful as the prompts behind it. Some tools use prompts you type in by hand. Some generate synthetic prompts. Some build from real search demand, which is closer to how buyers actually phrase things. Ahrefs Brand Radar, for instance, draws on search-backed prompts rather than invented ones. HubSpot's AEO tool suggests prompts based on what it already knows about your business and buyers from your CRM. A polished dashboard built on a generic prompt list will give you confident answers to questions nobody is asking.
2. How many times does it run each prompt? Given what the SparkToro data shows about variance, a single run is noise. Credible tools run each prompt several times across a measurement window and report a mention rate as a distribution, not a one-off result. As one measurement guide bluntly puts it, running each prompt once per cycle is standard in most off-the-shelf tools, and it is a failure mode: when a single-run number shifts between cycles, you cannot tell a genuine change from normal answer variation. If a tool reports off one snapshot, distrust the number.
3. How does it collect the answer? Some tools query the model's API. Others capture the answer a real user would see in the product interface. The front-end approach reflects actual exposure more accurately, but it costs more to operate, which is part of what you are paying for at the higher tiers. API results and front-end results can differ, so two tools using different methods will not match.
4. How often does it run? Daily, weekly, monthly. Cadence is partly a budget decision, because every run costs API or capture spend, and partly a data-quality one, because thin sampling produces shaky trends. A sensible pattern is frequent polling for your highest-intent prompts and lighter polling for the long tail.
Change any one of these four levers and the headline number changes. That is why cross-tool comparisons are close to meaningless, and why the only fair comparison is a tool against itself over time.
A short checklist before you trust a number
If you are evaluating a tool, or already paying for one, ask the vendor four direct questions:
- Which prompts are you running, and where did they come from?
- Which engines do you cover, and which do you miss?
- How many times do you run each prompt, and over what window?
- How do you score a mention, and can I see the underlying answer text?
A good tool answers all four plainly and will show you the stored responses behind every score. If a vendor cannot tell you the prompts, the engines, the frequency, and the scoring model, you are paying for a black box. This skepticism is mainstream now, not contrarian; marketers across the industry are questioning whether these tools deliver what they promise, and the honest vendors are the ones upfront about where their data has gaps.
Three things a "visibility score" quietly blurs
Even a well-built number hides distinctions that matter for what you do next.
Appearing is not the same as being recommended well. A brand can show up in nearly every answer and be described as overpriced and unreliable. Zero visibility and bad visibility look identical if all you track is appearance rate. Sentiment and accuracy are separate signals, and in regulated industries an AI confidently misattributing something to your brand is a compliance problem, not a marketing miss.
A mention is not a citation, and a citation is not a click. Being named in an answer builds association. Being cited as a source can drive a visit. Neither guarantees the other, and most dashboards collapse them.
Visibility is not traffic. This is the one most teams conflate. AI visibility tools estimate whether you show up in answers. Your analytics tells you who actually clicked through to your site from an AI tool. They answer different questions and they are measured in completely different ways, one by simulation and one by real sessions. We cover the traffic side, and why even that is incomplete, in our guide to tracking AI referral traffic in GA4. You want both, and you should never read one as a proxy for the other.
How to actually use the data
None of this means the tools are useless. It means they are instruments with a known margin of error, and instruments like that are valuable when you read them correctly.
Use them for trends, not absolutes. Watch your mention rate move over quarters, on a prompt set that reflects how your buyers really ask. Use them for competitive gaps, the prompts where a competitor consistently appears and you do not, because those gaps are content briefs in disguise. Use them to catch sentiment and accuracy problems early. And pair the picture with manual spot-checks, because running ten real buyer prompts yourself, by hand, remains one of the clearest windows into how AI describes you.
What you should not do is treat a single score as truth, compare two tools' numbers as if they were the same measurement, or report a decimal-point figure to leadership as if it were a fact. The number is a model. Good marketing comes from knowing that, and reading it accordingly.
