AI Crawler Checker · Website
AI Crawler Access Checker (GPTBot, ClaudeBot, PerplexityBot…)
See whether ChatGPT, Claude, Perplexity and Gemini crawlers can read your site.
- 100% free
- No sign-up
- Live data from our servers
How it works
How to use AI Crawler Checker
- 1
Enter your domain (or a specific page to test that path).
- 2
Review which crawlers are allowed or blocked and which rule decided it.
- 3
Follow the recommendation, or copy the suggested robots.txt snippet.
About
About this tool
AI assistants and answer engines find content in different ways. Some crawlers collect pages for training models (GPTBot, ClaudeBot, CCBot), some build search indexes that let assistants cite you (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and some fetch a page only when a user asks about it (ChatGPT-User, Claude-User, Perplexity-User). Your robots.txt decides which of them may visit.
This checker downloads your robots.txt and evaluates it for 18 crawlers using the official matching rules from RFC 9309: the crawler's own user-agent group wins over the * group, the longest matching path wins, and Allow beats Disallow on a tie. For each bot you see Allowed or Blocked and the exact rule that decided it.
For answer engine optimisation (AEO) and generative engine optimisation (GEO), the usual advice is to keep AI search and user-triggered bots allowed so assistants can read and cite you. Blocking training crawlers is a separate business choice, and the suggested snippet shows how to do both.
Frequently asked questions
Free proposal
Want to rank higher? We'll turn these findings into an SEO, AEO and GEO plan for your site.
No obligation. A real strategist replies within 24 hours.
Keep going
More free tools
Done-for-you
Let's get you ranking on Google and AI search
Free tools show you the problem. The T3 team fixes it, with a clear plan and pricing up front.