If you've hired anyone to help with AI search visibility lately, you've probably seen a dashboard with a number on it. A position. A score. A rank that goes up or down week to week, next to your logo and a few competitors.
It looks like something you already understand. It looks like a Google ranking.
That's the problem.
AI answer engines like ChatGPT, Google AI Overviews, Gemini, and Perplexity don't return a ranked list of ten results the way Google search used to. They generate an answer, and they either mention your business somewhere in that answer or they don't. There's no page one. There's no slot three. So when a report hands you a single number and calls it your "AI ranking," that number had to come from somewhere, and it's worth understanding where before you treat it as fact.
A number with no method behind it is decoration, not data.
This isn't a knock on every tool or every agency doing this work. Some of it is genuinely useful. But a lot of what's being sold right now takes a metric built for a different era of search and repackages it to look familiar, because familiar sells easier than honest. Here's what to ask before you take a dashboard number at face value.
01 /"What exactly does 'position' mean here?"
Ask this plainly and expect a plain answer. If your business shows up at "position 6" in an AI answer, what does that mean when the AI didn't return six things? We've watched a vendor report a specific position for a query that, when tested independently, returned nothing about the business at all, on every platform they claimed to be tracking. If nobody can walk you through how the number was calculated, treat it as decoration, not data.
02 /"Is this an average across platforms, or broken out by each one?"
ChatGPT, Google AI Overviews, and Gemini don't agree with each other. They pull from different sources and weight different signals. One might lean heavily on your Google Business Profile, another might care more about specialty content on your website, another might barely use your GBP at all. A single blended number across all of them hides the thing you'd actually want to act on. If your report only gives you one average, ask for the platform-level breakdown before you decide anything based on it.
03 /"How many times was this tested, and over what window?"
AI answers change. Ask the same question twice in the same week and you can get a different result, not because anything about your business changed, but because the underlying model is not deterministic in the way a search engine's index is. A single snapshot dressed up as a stable number is misleading no matter how confident the dashboard looks. Ask how often the test runs and whether they've seen the number move between tests. If they haven't checked, that's your answer.
04 /"What tool is actually pulling this data?"
A lot of AEO reporting right now runs on top of tools originally built for traditional SEO rank tracking, with AI visibility bolted on as a feature. That's not automatically bad, but it means the underlying tool may be forcing your results into a "position" format because that's the format it was designed around, not because that's how the AI actually responded. Ask directly. If nobody can name the tool or explain what it's measuring under the hood, that's worth knowing.
05 /"Can I see the actual response, not just the score?"
This is the simplest gut check there is. Ask for the raw AI answer that produced the number, word for word. If your business is genuinely mentioned, you'll see it right there in the text. If what you get back is a chart with no receipts, ask again. A vendor confident in their own methodology should have no problem showing you the source.
06 /"What's the goal I should replace 'position' with?"
Here's the honest version of what actually matters, especially if you're a local or founder-led business: whether AI engines can find and verify your business at all, whether independent sources online back up your claims about your business (reviews, directories, credentials), and whether you've published enough specific content that AI has something real to cite you for. That's a foundation you can build. A number with no method behind it is not.
The bottom line
None of this means every AI visibility report is worthless, and it doesn't mean automation is the enemy. Tools have a real place here, and testing this by hand for every client, every week, isn't realistic for most agencies at scale. The problem isn't the tool. It's when a number gets handed to you with more confidence than the method behind it deserves, and nobody's willing to explain how it was built.
Ask the questions. If you get straight answers, you've probably found a partner worth trusting. If you get a shrug, you've learned something too.