AI Can’t Make Up Its Mind: Rethinking How We Measure Brand Visibility

by | Mar 31, 2026 | Branding & Logo Development, Marketing & Business Development | 0 comments

AI “rankings” look impressive on a dashboard—but new research shows they’re basically a slot machine. What is useful is how often your brand shows up in AI answers, not whether you’re “number 1.” In a January 2026 study, Rand Fishkin and SparkToro, in partnership with Gumshoe.ai, analyzed nearly 3,000 AI responses to see how consistent brand recommendations really are.

Below is a plain‑English breakdown of what that means and what to actually do about it in your marketing.

 

1. What the SparkToro + Gumshoe Study Discovered (In Plain English)

Fishkin’s team at SparkToro, working with Gumshoe.ai, set out to answer a simple question:

Are AI results stable enough that it makes sense to “track rankings” in tools like ChatGPT, Claude, or Google’s AI answers?

They asked 600 people to run 12 different prompts through three AI tools—ChatGPT, Claude, and Google’s AI Overviews/AI Mode—almost 3,000 times. Then they normalized the responses so they could compare the lists of brands and products.

Here’s what they discovered in everyday language:

    • The lists keep changing
      Ask the same AI the same question over and over and you almost never get the exact same list twice.
    • The order is all over the place
      Even when the same brands appear, their order is essentially shuffled from one answer to the next.
    • Even the number of results changes
      Sometimes you get a short list (2–3 options), sometimes a much longer one (7–10+). There is no reliable “top 10.”

According to SparkToro’s analysis, statistically:

    • Less than a 1 in 100 chance of seeing the same list twice.
    • Less than a 1 in 1,000 chance of seeing the same list in the same order.

In other words: AI tools are not behaving like old‑school Google search results. They are not fixed rankings; they generate slightly different, reasonable‑looking answers every time. That blows up the idea that you can reliably track “your position” in AI.

2. Why This Makes “AI Rank Tracking” Misleading

Despite this, a lot of tools and agencies are now selling reports like:

    • “You’re #2 in ChatGPT for ‘best accountant in Edmonton’”
    • “Your average rank in Google AI for ‘top SaaS CRM’ is 4.3”

In light of the SparkToro/Gumshoe findings, we can say with confidence that those specific position numbers are mostly noise.

Here’s why this matters for you as a business owner:

    • “Position 1” might just be luck
      If you ran the same question 50 more times, your brand might vanish or bounce around from 7th to 3rd to nowhere at all. That “win” could be a fluke rather than a meaningful result.
    • You risk moving budget based on fake precision
      A report that says “we dropped from #2 to #5” sounds serious, but in an unstable system that shift may mean absolutely nothing.
    • It enables snake‑oil tactics
      A less‑than‑honest vendor could keep re‑running prompts until the AI happens to show you in a good position, screenshot it, and take credit.

The bottom line: any tool that sells you exact ranking positions inside AI answers should be treated with serious skepticism.

3. The Helpful Part: Visibility, Not Ranking

The good news is that the study doesn’t say AI is useless—it says we’re looking at the wrong metric.

While specific rankings bounce around, one pattern is much more stable:

Some brands show up again and again across many runs of the same type of question.

That repeat appearance is what I’m calling visibility.

Think of it this way:

  • If you asked ChatGPT “best marketing automation tools for small business” 100 times and your brand appeared in 70 of those answers, you’d have 70% visibility for that kind of question.
  • If a competitor appeared in 10 out of 100 answers, they’d have 10% visibility.

The exact order (1st, 3rd, 5th) is noisy. The fact that you show up in 70 out of 100 responses is not; it tells you:

    • The AI model “knows” your brand for that topic.
    • You’re in the short list more often than many competitors.

This is where AI can genuinely inform your strategy: not “What’s my rank?” but “How often do we show up at all when someone asks about our category?”

4. Why Prompts Make Things Messy (But Still Useful)

There’s another wrinkle: real people do not type the same prompt.

In Google Search, we got used to short keyword strings like “Edmonton divorce lawyer fees.” In AI tools, people write full questions and scenarios, like:

In AI tools, people write whole sentences and scenarios, like:

    • “My sister lives in Edmonton and is going through a difficult divorce, what are some trustworthy local law firms that won’t totally bankrupt her?”

    • “If I’m in Alberta and need a family lawyer for custody disputes, which firms have a good reputation?”

On the surface those prompts look very different, even though the intent is similar.

SparkToro’s follow‑up work with real human prompts showed:

    • People almost never phrase prompts the same way.

    • Even so, for a given intent (for example, “which headphones should I buy for travel”), the same core brands tend to show up again and again.

So yes, prompts are messy. But the AI still tends to draw from a fairly consistent pool of brands when the underlying intent is similar.

For you, that means:

    • You can’t hang everything on one single “magic” prompt.

    • You can look at whether you appear often across a bunch of different prompts that all point to the same kind of need.

5. What You Should Actually Do (Action Plan)

Based on this research and my own experience working with small and mid‑sized businesses, here’s how I think owners should respond.

Step 1: Stop Chasing “Position 1 in AI”

  • Don’t set KPIs like “Get to rank #1 in ChatGPT for [keyword].”
  • Don’t reward your team or agency for screenshots of one great result.
  • Don’t let “we’re #3 in AI this month” drive major budget decisions.

Treat those as interesting anecdotes, not reliable metrics

Step 2: Start Thinking in Terms of “Being in the Shortlist”
Your real goal: when someone asks an AI tool about your category, you want your brand to be one of the 3–10 options that show up often.

Practical moves:

    • List your key buying situations
      For example: “best hot tub for a cold climate,” “marketing agency for ecommerce brands,” “accountant for small construction businesses,” etc.

    • For each situation, brainstorm 10–20 ways a real human might ask the question
      Use language from emails, sales calls, reviews, and live chat transcripts.

    • Occasionally test these in AI tools (manually)
      Don’t obsess over the exact order. Just note: do we show up at all, and how often compared to the brands you usually compete with?

You’ll quickly see whether you’re in the conversation or invisible.

Step 3: Build Real‑World Signals That AIs Can See

AI tools base their answers on what they can “see” across the web. To show up more often, you need strong, clear signals that your brand is a good answer for your topics.

Focus on:

    • Clear, specific content on your site
      Create pages and posts that match the problems and questions people actually ask, written in normal language—not just keyword strings.

    • Proof of expertise
      Case studies, testimonials, reviews, media mentions, and podcast appearances show that other humans trust you.

    • Consistent, up‑to‑date information
      Make sure your services, pricing structure, locations, and specialties are accurate across your site and major profiles.

      Being mentioned in trusted places
      Industry directories, local media, niche blogs, and associations all add evidence that you’re a credible option.

Think of it like this: if the internet were a courtroom, you want lots of credible witnesses saying, “Yes, this company is a solid answer for that problem.”

Step 4: If You Use an AI Visibility Tool, Ask These Questions

If a vendor is pitching you an AI tracking platform, don’t just look at the shiny charts. Ask:

  • Do you track visibility (how often we appear) or just rankings?

  • How many times do you run each type of prompt to smooth out randomness?

  • How do you choose or generate prompts—are they realistic for our audience?

  • Do you have public, documented research on how stable your metrics are?

If they dodge these questions or won’t explain their methods, be cautious. You wouldn’t trust a financial report that never explains how the numbers are calculated; treat AI metrics the same way.

Step 5: Use AI Insights As One Input, Not The Gospel
AI visibility is just one piece of the puzzle. Use it alongside:

    • Website analytics (Are people visiting and converting?)

    • Search performance (Are you findable in regular Google search?)

    • Sales conversations (Are customers saying “We heard about you from [X]?”)

    • Reviews and word of mouth

AI can help you see whether your brand is “in the mix,” but it should not replace real‑world customer data and common sense.

6. The Simple Takeaway
If you take nothing else from this:

    • AI rankings (who’s #1, #2, #5) are too random to trust.
    • What matters is whether you consistently show up at all when someone asks about your kind of product or service.
    • The way to improve that is the same as good marketing has always been: be clearly positioned, visibly credible, and well‑documented online.

If you want help turning this into action for your specific niche, tell me what industry you’re in and I can suggest 10–15 concrete prompts and content ideas tailored to your business.

All statistics in this post about AI response consistency come from SparkToro and Gumshoe.ai’s 2026 study on AI brand recommendations, which you can read here: https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility/