Where does robots.txt go?

At the root of the domain, for example https://www.example.com/robots.txt. Each subdomain needs its own file.

Does Disallow remove a page from Google?

No. It stops crawling, but a blocked URL can still be indexed if other sites link to it. To remove a page, allow crawling and add a noindex tag instead.

Does Google respect Crawl-delay?

No. Google ignores Crawl-delay; Bing and some other crawlers use it. Google adjusts its crawl rate automatically based on how your server responds.

Should I block AI crawlers?

It depends on your goals. Blocking training crawlers keeps content out of future models; blocking AI search and user-triggered bots can stop assistants from citing you. Many businesses allow search and block training.

Robots.txt Generator · SEO & AI search

Robots.txt Generator — With AI Crawler Presets

Create a robots.txt with sitemap, disallow rules and per-bot AI crawler controls.

100% free
No sign-up
Runs in your browser

Start from a preset

Your site URL

Used for the sitemap line in presets.

Default for all other crawlers

Disallow paths

One per line. * and $ wildcards work.

Allow paths

Exceptions inside disallowed folders.

Crawl-delay (seconds, optional)

Bing and some others respect it; Google ignores Crawl-delay.

Sitemap URLs

Full URLs, one per line.

Per-crawler rules

Search engines

GooglebotGoogle Search
BingbotBing / Copilot

AI search

OAI-SearchBotChatGPT search
Claude-SearchBotClaude search
PerplexityBotPerplexity search
AmazonbotAmazon

AI assistants (user-triggered)

ChatGPT-UserChatGPT (user fetch)
Claude-UserClaude (user fetch)
Perplexity-UserPerplexity (user fetch)

AI training

GPTBotOpenAI training
ClaudeBotAnthropic training
anthropic-aiAnthropic (legacy)
Google-ExtendedGemini training
Applebot-ExtendedApple AI training
Meta-ExternalAgentMeta AI
CCBotCommon Crawl
BytespiderByteDance
cohere-aiCohere

robots.txt

User-agent: *
Allow: /

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Amazonbot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
User-agent: Bytespider
User-agent: cohere-ai
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Upload as /robots.txt at the root of your domain (e.g. https://www.example.com/robots.txt). A crawler with its own group ignores the * group, so we repeat your path rules for any bot you explicitly allow. robots.txt asks crawlers politely; it doesn't secure private pages. Built in your browser.

How it works

How to use Robots.txt Generator

Pick a preset and enter your site URL.

Adjust paths, sitemaps and individual crawler rules.

Download robots.txt and upload it to the root of your domain.

About

About this tool

robots.txt is a plain-text file at the root of your site that tells crawlers which parts they may visit. It's where you point search engines to your XML sitemap, keep them out of admin or cart pages, and, increasingly, decide which AI crawlers can use your content.

Start from a preset: allow AI search but block AI training, block all AI crawlers, allow everything, or WordPress defaults. Then fine-tune the default rule, disallow and allow paths, crawl-delay and sitemap URLs, and set each of 18 search and AI crawlers to default, allow or block. The file updates live and downloads ready to upload.

One subtlety the generator handles for you: a crawler with its own group ignores the * group entirely, so any bot you explicitly allow gets your path rules repeated. And remember robots.txt is a public request, not security; protect private pages with a login.

Frequently asked questions

Free proposal

Want to rank higher? We'll turn these findings into an SEO, AEO and GEO plan for your site.

No obligation. A real strategist replies within 24 hours.

Keep going

More free tools

All free tools

AI Crawler CheckerNewSee whether ChatGPT, Claude, Perplexity and Gemini crawlers can read your site.LLMsT3NewTurn your sitemap into an llms.txt file that guides AI tools to your best pages.Meta Tag GeneratorGenerate title, description, canonical, robots, Open Graph and X tags with a live preview.SEO CheckT3PopularScan any page for title, meta, headings, schema, links and 30+ on-page SEO checks.

Done-for-you

Let's get you ranking on Google and AI search

Free tools show you the problem. The T3 team fixes it, with a clear plan and pricing up front.

See our services Run a free website audit