Traditional Googlebot crawling is falling while AI retrieval bots climb. Dan Kurtz of Miss Pepper AI reads it off his own server logs, and his conclusion is not that SEO is over. It is that we have done this before. We have been multi-purposing content into infographics, reels and press releases for twenty years because humans have preferred content types. Bots have one too.
As SEOs and content marketers, we’ve been doing that for decades. We call it multi-purposing. Bots have a preferred content type. Technically, they’re just an additional customer.
— Dan Kurtz, 37:42
The practical version is a plain-text or markdown file sitting alongside the HTML page. Dan’s case for doing it is that the risk is asymmetric:
I would absolutely spend the time to go ahead and do that and invest in it, because the worst thing that can happen is they pass it. They fly over it.
— Dan Kurtz, 37:42
Then the step almost everyone skips. “That’s just a matter of directing them to it next time they come to the website and saying, hey, this file exists.” We took that literally and published machine-readable companions across twelve episodes, linked from every post. One thing we found by accident: our own flagship site had been returning 429 to GPTBot on every page.
Not everyone agrees. Malte Landwehr makes the opposite case, that a second URL carrying the same content is not cost-free, and would serve markdown by user agent at the same URL instead. Two practitioners, two defensible positions, which is usually the sign that the question is open rather than solved.
Traditional Google bots down, I think something like sixty percent crawling sites, but AI bots are up eighty percent for doing the crawling and retrieval. It’s just looking at server logs and stuff like that, you know, got my own sites that I can see those kinds of tests on.
— Dan Kurtz, 6:59
He frames it as his own telemetry rather than an industry benchmark, and that distinction matters. The independently verified figure is Cloudflare’s: automated traffic overtook human traffic on the web in June 2026.
None of this appears in Search Console. Bot requests are not impressions. If you want to know which crawlers actually reached you and what they got, the answer is in your access logs, and we wrote up the log-analysis method for auditing AI bot traffic as a standing SOP.
If you go ahead on your hosting provider and you go ahead and you reissue a new SSL certificate, it will force all of the AI bots to recrawl your stuff all over again... I don’t know what the cadence is, I don’t know what the frequency is.
— Dan Kurtz, 38:09
Treat that as a hypothesis worth testing rather than a tactic. He says so himself.
Everything we do for SEO, I’ve got a dashboard that’s got an API. The thought leadership platform for the exec, that’s got an API. The call and appointment setting platform, that’s got an API. So now all of these all talk to each other and they’re all passing baton data between each other.
— Dan Kurtz, 26:18
Sales talks to SEO talks to marketing talks to appointment setting. He expects to publish results on what that does to a business around Q1 2027.
One component is an A/B loop that hands Microsoft Clarity and SEO Gets data to Claude and lets it rewrite the page. Worth a warning before you build the same thing: Clarity’s regex filters silently return nothing, and Search Console and GA4 run a third engine again. Our pattern compatibility checker tells you which of the three will actually accept a given filter, and the regex library has the patterns themselves organised by the question they answer.
Deliverables in SEO means that you’re not paying attention. It’s you’re focused on making sure you hit that quota every month.
— Dan Kurtz, 13:25
His alternative is an enablement layer: AI first, humans working on top of it, and no scope creep pushed down onto whoever is doing the work.
| Time | What happens |
|---|---|
| 0:14 | Miss Pepper AI’s origin, and the 2012 Google penalty that killed the first version |
| 2:15 | Seth Godin’s Linchpin, then Black Hat World |
| 6:59 | Googlebot down 60%, AI bots up 80%, read off his own server logs |
| 13:25 | “Deliverables in SEO means that you’re not paying attention” |
| 26:18 | The agentic flywheel: an API between every function |
| 37:42 | Bots as a customer segment with a format preference |
| 38:09 | Linking to the file, and the SSL reissue trick |
| Entity | What it is |
|---|---|
| Miss Pepper AI | Dan’s agency and product, built on an AI enablement layer beneath human work |
| Data for LLMs | His approach to publishing structured data straight to the models |
| Microsoft Clarity | Behaviour analytics feeding his A/B rewrite loop |
| SEO Gets | Search Console data layer in the same loop |
| Cloudflare bot report | The independent source for the crossover claim |
| Seth Godin, Linchpin | Named as the turning point early in his career |
Every link above goes somewhere different. These three are the ones not already mentioned in the conversation above.
Kurtz, Dan. Interviewed by Jeremy Rivera. “Bots Are Just Another Customer.” The Unscripted SEO Interview Podcast, 3 August 2026. https://unscriptedseo.com/dan-kurtz-bots-as-a-customer-segment-markdown-agentic-flywheel/
Dive into the intricate world of backlink evaluation with Mark A Preston and Kasra Dash. The age-old question remains: How often should you assess...
Guest: Lorraine Ball, Host of More Than a Few Words Podcast Experience: 19 years running a digital marketing agency, sold in 2021 Current Focus:...
Dive into the dynamic world of affiliate marketing with Mark A Preston and Edd Dawson as they unravel the strategies behind thriving in large...
