Your AI visibility score is probably a coin flip
Most tools ask the model once and report the answer as a rank. But LLM output is a probability distribution, not a leaderboard. Why single-shot measurement is noise, and what a defensible number looks like.
Most AI visibility tools will tell you that you rank third for a query. Ask the model the same question again and you might rank first, or not appear at all. The number was never a rank. It was one sample from a probability distribution, reported as if it were a fact.
This is the uncomfortable thing about an entire young category: a lot of the numbers being sold are single observations of a random process. If you are going to spend budget on this channel, it is worth understanding why, and what a defensible number looks like instead.
Why the same question gives different answers
A language model generates text one token at a time. At each step it produces a probability distribution over possible next tokens and picks one. How it picks is governed by a setting called temperature. At temperature zero it always takes the most likely token and behaves close to deterministically. Above zero it samples, which is what makes output feel varied and natural rather than robotic.
Consumer assistants run above zero. That is a product decision, not a bug. But it means the answer to “best pressure washer in India” is not a fixed list. It is a distribution over lists.
Retrieval adds a second layer of variance. Modern answer engines expand one question into many back-end searches, a technique Google has described as query fan-out for its AI Mode. Different sub-queries surface different pages, different pages survive reranking, and the model synthesizes from a slightly different evidence pile each time.
What a defensible number looks like
If the underlying reality is a distribution, the honest unit of measurement is a probability with a stated confidence, not a position on a leaderboard. The claim should read: this brand appears in this answer 40 percent of the time, at an average position of three, on this engine, on this model version, measured on this date, from this many samples.
The statistics are not exotic. For a proportion, the uncertainty shrinks roughly with the square root of the sample count. At one sample you have nothing. Around seven samples you can separate a brand that dominates from one that rarely appears. At twenty you have an estimate tight enough to compare brands within a category and to detect real movement over time.
None of this is difficult. It is just more expensive than asking once, which is precisely why so few tools do it.
The version problem underneath
There is a second source of drift that is easy to miss. The model you measured last month is not the model you are measuring today. Providers update weights and system behaviour continuously, and an update can reshuffle which brands a model favours.
This creates a genuine attribution trap. Your visibility improves, and you credit the content you shipped. It may equally have been a model update that had nothing to do with you. Without version stamps on every measurement, there is no way to tell the two apart, and you end up optimizing against noise.
Three questions to ask any vendor
How many times do you sample each prompt, and do you publish it? If the answer is one, or if they will not say, the number is not a measurement.
Can I see the raw answer behind this score? Every derived number should open to the exact prompt, the verbatim response, and the timestamp. If you cannot audit it, you are being asked to trust a black box in a category whose entire value is transparency.
Do you stamp the model version? Without it, month-over-month comparisons are not comparable.
Why this is not just methodology pedantry
Because decisions get made on these numbers. A team sees a score drop and rewrites a page that was fine. A team sees a score rise and doubles down on content that did nothing. In a channel where you cannot see clicks and cannot see referrers, the measurement is the feedback loop. If the loop is noise, the strategy is noise.
We publish our sample sizes because we would rather show a wide confidence interval than a clean number we cannot defend. When our India AI Visibility Index first went out with too small a sample, we rebuilt it at twenty runs per engine per category rather than publish something that contradicted our own methodology.
Common questions
Why do I get different answers from ChatGPT for the same question?
Consumer assistants generate text by sampling from a probability distribution at a non-zero temperature. That randomness is deliberate, it makes answers feel natural. The side effect is that the same prompt can return different brands on different runs.
How many times should you sample an AI visibility check?
One run tells you almost nothing. Around seven runs distinguishes large differences. Twenty runs per engine gives a reasonably tight estimate for category-level comparison. Whatever the number, a credible tool publishes it alongside the result.
What is the right metric for AI visibility?
A probability, not a rank. The honest unit is 'this brand appears in this answer X percent of the time, at average position Y', reported per engine and stamped with the model version and date.
Does a higher AI visibility score mean more traffic?
Not necessarily. Most AI answers end without a click, so presence in an answer influences demand without producing a measurable visit. Treat visibility as share of influence, not as a traffic forecast.
Our full sampling approach and scoring curve are published on the methodology page. If you want to see the variance yourself, run a free scan and then run it again.
One email when a new post drops. No newsletter fluff, no 'top-of-mind' spam.
No open tracking. Unsubscribe with one click. Never sold.