What are AI crawlers actually served?
Your settings might say “welcome” while your firewall blocks AI at the door. This test requests your site as ChatGPT, Claude, Perplexity and Google — and shows you exactly what each one gets back.
We test the door, not the doormat
robots.txt is the doormat with a note on it. We check whether the AI crawlers can actually get through the door — measured live, the moment you run the test.
We knock on your door as each AI
We fetch your homepage as each of seven AI crawlers — the ones behind ChatGPT, Claude, Perplexity, Google and Bing — then once more as an ordinary browser, so we have a baseline to compare against.
We read what actually came back
Not just the status code. A “200 OK” can hide a blank “checking your browser” challenge page. We inspect the real content each crawler received and classify it honestly.
We cross-check it against robots.txt
The headline finding: when your robots.txt says “come in” but your security layer quietly serves a challenge instead — the AI gets nothing, and you’d never know from robots.txt alone.
The gap that quietly costs you AI traffic
Every rival tool stops at robots.txt
robots.txt is only a polite note asking crawlers in or out. It says nothing about what your server, CDN or firewall actually hands back. We measure the real thing.
Status code is not content
A firewall can answer “200 OK” with an empty human-verification shell and zero words. Status-only checkers call that healthy. AI sees a blank page — and skips you.
The GPTBot vs OAI-SearchBot trap
Blocking GPTBot only stops model training. Blocking OAI-SearchBot stops ChatGPT citing you in answers. Most sites confuse the two — we spell out exactly what each block costs.
Fair questions
How is this different from a robots.txt checker?
Robots.txt only states an intention. This test acts on it: we actually visit your site pretending to be each AI crawler and report what your server really serves them — real content, a security challenge, or an error. That gap between “allowed on paper” and “blocked in practice” is invisible to every robots.txt-only tool.
Will this affect or change my website?
No. It only reads your homepage and robots.txt a handful of times, the same way an AI crawler would. Nothing is changed, installed or stored on your site.
Why would my site block an AI crawler without me knowing?
Usually a security layer like Cloudflare. Its default bot protection can serve AI crawlers a “verify you are human” challenge page instead of your content — even when your robots.txt clearly allows them. It’s protecting you from bad bots, but it catches the good ones too.
Do I need to enter my email?
No. This test is deliberately open — run it as many times as you like. If you want the full picture across all five AI-visibility categories, that report is one click away.