Skip to main content
Contact
Journal

Fix in public, month 2: the tracker broke. Here's what I'm doing about it.

· 6 min read · By Max Muncy

Month one of this series promised that month two would deliver the baseline citation numbers from a thirty-prompt, six-engine AI-search tracker. The plan was to publish those numbers — wherever they landed — as the floor we’d be climbing from. I ran it this week. The automated portion failed. Here’s the honest version of what happened, what I’m doing about it, and what shipped this month regardless.

What was supposed to happen

The tracker is a Python script that queries thirty target prompts across six AI search engines (ChatGPT, Perplexity, Google AI Overview, Claude, Gemini, Microsoft Copilot) and records whether Mainsail is named in the answer. Two of the six engines can run automated — Perplexity (no login required) and Google’s AI Overview (also unauthenticated). The other four require logged-in sessions and have to run manually.

The plan for month two: run the automated portion (~15 minutes), have me run the manual portion (~30 minutes spread across four engines), aggregate the data, publish the baseline numbers. Real ones, even if they were near-zero — especially if they were near-zero.

What actually happened

The automated run finished in 13 minutes with zero successful citations recorded across all 24 prompts on either engine. Not zero citations — zero successful queries. The breakdown:

Perplexity: every single one of the 24 prompts returned the same error — “textarea not found: Page.wait_for_selector: Timeout 15000ms exceeded.” The Playwright script targets a <textarea> element on the Perplexity homepage that’s been stable since the script was written. Apparently it isn’t anymore. Perplexity changed their query-input UI sometime in the last few weeks. The selector needs to be rewritten.

Google AI Overview: every single prompt returned [CAPTCHA]. Google’s automated-traffic detection caught the script. Headless Chromium hitting Google’s search endpoint at a 5-second pace is enough to trigger their /sorry/ challenge page. The script paces requests to avoid this, but Google has gotten stricter.

So the dashboard I was going to publish is twenty-four rows of error messages. Not the baseline numbers I promised.

The manual portion — ChatGPT, Claude, Gemini, Copilot — I haven’t run yet either. Going to be honest: I’d planned to do it this week and got pulled into rewriting the homepage instead (more on that below). Manual checks coming next week.

What this means and what it doesn’t mean

What it doesn’t mean: Mainsail is or isn’t being cited in AI search. The tracker tells us nothing about citation rates this month because the tracker didn’t actually check.

What it does mean: the tooling I built to measure progress in public broke between month one and month two. That’s not a small thing. The entire premise of the fix-in-public series is that the numbers get published as the work happens — and a broken tracker is a broken accountability mechanism.

The honest read: building an automated AI-visibility tracker is itself a moving-target engineering problem. The engines change their UI. Google tightens their bot detection. Perplexity ships a redesign. Anyone selling you “AI search rank tracking” as a stable monthly subscription product is either constantly maintaining the tooling or quietly under-delivering. There’s no third option.

What I’m doing about it

Three things, in order of priority:

1. Rebuild the Perplexity automation against the new UI. Should be a 30-45 minute fix — update the selector, re-test, re-run. Probably ships this weekend.

2. Switch the Google AI Overview check to a different approach. The Playwright-against-Google route is increasingly fragile. The alternative is DataForSEO’s SERP API, which costs ~$0.004 per query and explicitly surfaces the AI Overview block. Thirty prompts × $0.004 = $0.12 per monthly run. That’s cheaper than the engineering time to keep maintaining a brittle Playwright script. I’m migrating Google to this path.

3. Run the four manual engines this week while the automation gets fixed. ChatGPT, Claude, Gemini, Copilot. Paste each prompt, record the Y/n/- coding, save the excerpts. Manual is slower but it’s the floor — it always works. The point of automating the easy two was to save time, not to replace measurement entirely.

If all three land before month three’s update, the next post has real numbers across all six engines. If only some land, the post has partial numbers and explains what’s missing. Either way the cadence keeps going.

What shipped regardless

The tracker isn’t the only thing that happened this month. The work that did get done:

  • How to optimize a Google Business Profile (the 2026 checklist) — published today. The single highest-leverage local-SEO move available, free, almost nobody’s doing it.
  • SEO agency for small businesses: how to actually evaluate one — published today. The inverse-framework for evaluating an SEO agency: not what they pitch you, but what they ask you in discovery.
  • The homepage rewrite that’s live as of today. New H1 (“Be the first business Greenville customers find — in Google and in AI”), a trust strip under the hero with real credibility signals (5.0★ Google rating, the Knightstown 9→150 case, sub-1s page load, Astro + Cloudflare stack), and a closing bookend that reinforces the same outcome promise the hero opens with. The previous hero was positioning-led (“Get cited by AI / Show up in Google / Win Greenville”) — distinctive, but didn’t earn the scroll. The new version is pain-led, and the data over the next 30 days will tell us whether it converts better.

The homepage rewrite came out of the conversion-data Journal piece from last week, where we worked through why ranking #1 for our hardest commercial keyword wasn’t generating buyers. The diagnosis pointed at intent (the query isn’t really buyer-shaped) but also at funnel friction (1.36 pages per visit on Cloudflare — most visitors weren’t scrolling past the hero). The rewrite addresses both: pain-led copy gives the buyer a reason to scroll, and a closing bookend repeats the offer so the page closes the way it opens.

I’d rather have the citation numbers AND the homepage rewrite. But if the tooling broke and I had to pick one, the homepage rewrite was the right pick — it’s a measurable change in the funnel that affects buyer conversion this month, not a measurement of a state that may or may not have moved.

The harder question I keep avoiding

The original premise of the Fix in Public series was: Mainsail doesn’t show up in AI search, here’s how we fix it, here are the numbers as we climb. The implicit assumption was that the climb would be measurable from week one.

Two months in, what I’ve actually learned:

  1. The climb is real but not where the simple metrics suggest. We have organic rank, we have GBP visibility, we have content shipping. The conversion outcomes are smaller than the rank improvements would predict — because most “high-intent commercial” queries aren’t actually high-intent in the way keyword tools suggest.
  2. The measurement layer matters more than I expected. A broken tracker isn’t a measurement problem; it’s a credibility problem. If I can’t measure citation rates, I can’t honestly tell prospects what we’d improve for them. That’s a problem to solve in the next month, not later.
  3. Showing up in AI search is roughly the right goal, but the leading indicators are GBP completeness, Reddit/YouTube presence, schema completeness, and entity disambiguation — not citation counts in week one. Citations are the lagging indicator. We’ve been chasing the lagging indicator and underweighting the leading ones.

Month three will publish whatever the citation tracker delivers when the tooling works. It’ll also probably reframe what “winning” looks like in this series. Not abandoning the fix-in-public arc — it’s been the most useful discipline of the year — but tightening what gets measured each month against what actually predicts the outcome.

What’s planned for month three

A short list:

  • Working tracker. All six engines. Real Y/n/- data for all 30 prompts.
  • Full thirty-prompt × six-engine matrix published. Not just aggregate stats. The full table. (This was the move I deferred this month per the soft-publication advice from the query-intent piece — at month three the matrix becomes useful as a comparison data point, not just a vulnerability surface.)
  • The query-intent retargeting numbers. Last week’s piece committed to push /services/local-seo to rank for buyer-shaped queries (“seo company in greenville,” “seo company near me”) rather than category-shaped ones (“greenville seo”). Month three reports what moved.
  • The homepage rewrite’s conversion data. Cloudflare analytics on the new hero, trust strip, and closing bookend. Specifically watching: pages-per-visit (currently 1.36), Loom-audit requests per visit, and any movement in inbound contact volume.
  • The Wikidata foothold — overdue from month one. Three independent third-party citations are the prerequisite (we wrote about this in the Apple Business Connect piece too), and I’m closer to having them this month than last.

If you’ve been following along — thanks. The next update is on the calendar for 30 days out. It’ll have data the tracker rebuilds let me publish.

— Max

Book Email Call