Skip to content
Free Tab Toolsnothing uploaded

Runs in your browser · Last updated 4 September 2026

Robots.txt Generator

Build a robots.txt with Allow, Disallow, Sitemap, and AI crawler groups. Copy and deploy — no account.

User-agent: *
Allow: /*
Disallow: /admin
Disallow: /drafts

Sitemap: https://example.com/sitemap.xml

How to use this tool

  1. Set Allow and Disallow paths for User-agent: *.
  2. Add your sitemap URL.
  3. Copy the file to the site root.
  4. Verify a sensitive path in the checker.

Allow vs Disallow

Google uses the longest matching rule. Allow: /admin/public beats Disallow: /admin for that subpath. Empty Disallow means “allow everything” for that group. Block GPTBot is the AI-crawler preset. llms.txt is a separate root file for pages you want cited. Page-level meta robots tags are a different file — they do not replace this robots.txt.

Frequently asked questions

Where does robots.txt live?
At https://your-domain/robots.txt — the site root, not a subdirectory.
Does Google honor Crawl-delay?
No. We omit it on the simple generator. Some other bots still read it.
How do I block GPTBot?
Use the Block GPTBot page. It emits a dedicated group so you do not lock out Googlebot. Pair with the llms.txt generator if you still want assistants to cite specific docs.
Can I test a path?
Yes — the robots.txt checker uses the same parser.
Is my file uploaded?
No.