If you've hired anyone to help with AI search visibility lately, you've probably seen a dashboard with a number on it. A position. A score. A rank that goes up or down week to week, next to your logo and a few competitors.
It looks like something you already understand. It looks like a Google ranking.
That's the problem.
AI answer engines like ChatGPT, Google AI Overviews, Gemini, and Perplexity don't return a ranked list of ten results the way Google search used to. They generate an answer, and they either mention your business somewhere in that answer or they don't. There's no page one. There's no slot three. So when a report hands you a single number and calls it your "AI ranking," that number had to come from somewhere, and it's worth understanding where before you treat it as fact.
A number with no method behind it is decoration, not data.
This isn't a knock on every tool or every agency doing this work. Some of it is useful. But a lot of what's being sold right now takes a metric built for a different era of search and repackages it to look familiar, because familiar sells easier than honest. Here's what to ask before you take a dashboard number at face value.
"What exactly does 'position' mean here?"
Ask this plainly and expect a plain answer. If your business shows up at "position 6" in an AI answer, what does that mean when the AI didn't return six things? It's not uncommon for a reported "position" to fall apart under independent testing: the business simply isn't mentioned when the same query is run cold. If nobody can walk you through how the number was calculated, treat it as decoration, not data.
"Is this an average across platforms, or broken out by each one?"
ChatGPT, Google AI Overviews, and Gemini don't agree with each other. They pull from different sources and weight different signals. One might lean heavily on your Google Business Profile, another might care more about specialty content on your website, another might barely use your GBP at all. A single blended number across all of them hides the thing you'd want to act on. If your report only gives you one average, ask for the platform-level breakdown before you decide anything based on it.
In my Tampa Bay acupuncture study, I asked ChatGPT, Google AI Overviews, and Gemini the same question: "best acupuncturist in St. Petersburg Florida." I got a different top recommendation from each one. A practice that only checked one tool would have walked away with the wrong picture.
"How many times was this tested, and over what window?"
AI answers change. Ask the same question twice in the same week and you can get a different result, not because anything about your business changed, but because the underlying model is not deterministic in the way a search engine's index is. A single snapshot dressed up as a stable number is misleading no matter how confident the dashboard looks. Ask how often the test runs and whether they've seen the number move between tests. If they haven't checked, that's your answer.
This isn't a hunch. In a study by SparkToro and Gumshoe.ai, 600 volunteers ran 12 identical prompts through ChatGPT, Claude, and Google's AI nearly 3,000 times. The odds of getting the same list of brands twice were under 1 in 100, and the odds of the same list in the same order were closer to 1 in 1,000 (SparkToro, 2026). What held steady was how often a brand showed up across many runs, not where it landed in the list.
"What tool is pulling this data?"
A lot of AEO reporting right now runs on top of tools originally built for traditional SEO rank tracking, with AI visibility bolted on as a feature. That's not automatically bad, but it means the underlying tool may be forcing your results into a "position" format because that's the format it was designed around, not because that's how the AI responded. Ask directly. If nobody can name the tool or explain what it's measuring under the hood, that's worth knowing.
"Can I see the actual response, not just the score?"
This is the simplest gut check there is. Ask for the raw AI answer that produced the number, word for word. If your business is mentioned, you'll see it right there in the text. If what you get back is a chart with no receipts, ask again. A provider confident in their own methodology should have no problem showing you the source.
"What's the goal I should replace 'position' with?"
Here's the honest version of what matters, especially if you're a local or founder-led business: whether AI engines can find and verify your business at all, whether independent sources online back up your claims about your business (reviews, directories, credentials), and whether you've published enough specific content that AI has something real to cite you for. That's a foundation you can build. A number with no method behind it is not.
When does AI visibility become a vanity metric?
AI visibility is a vanity metric when the number goes up and nothing else does. Watch for these signs:
- The questions being tracked aren't ones a paying customer would ask.
- The report shows a "rank" or "position" with no run count behind it.
- The score climbs, but calls, form fills, and bookings stay flat.
- Nobody checks whether what the AI says about you is correct.
Checklists can fool you too. One practice in my acupuncture study scored a perfect 7 out of 7 on my foundation checklist and wasn't named in any of the 27 tests. A good score on a checklist isn't the same as being recommended.
What counts as a good AI visibility score?
If the score is a citation rate, meaning the share of real customer questions where an AI tool names you, a business new to this work usually starts around 10 to 20%. 30 to 50% is solid, and 50% or higher is strong. Those ranges only mean something per question and per tool, measured over repeated runs. Here are the 7 metrics I use.
A single 0 to 100 "AI visibility score" with no questions or run counts behind it doesn't have a good or a bad. It's just a number.
The bottom line
None of this means every AI visibility report is worthless, and it doesn't mean automation is the enemy. Tools have a real place here, and testing this by hand for every client, every week, isn't realistic for most agencies at scale. The tools are fine. The problem starts when a number gets handed to you with more confidence than the method behind it deserves, and nobody's willing to explain how it was built.
There are providers out there who get this right: they'll show you the raw response, tell you flat out when a number can't be trusted yet, and would rather lose the sale than oversell a metric. Those are the ones worth building a relationship with.
Ask the questions. If you get straight answers, you've probably found a partner worth trusting. If you get a shrug, you've learned something too.
Sources: SparkToro and Gumshoe.ai, AI brand recommendation consistency study, 2026. Answer Visible, Tampa Bay acupuncture AI visibility study, July 2026.