Robots.txt Generator
Build a valid robots.txt with sitemap, crawl delay and AI-crawler options.
About this generator
Fill in the paths to keep crawlers out of, add your sitemap, decide what to say to AI crawlers, and copy a correctly formatted robots.txt. Generated in the page from your answers.
What robots.txt can and can't do
It's a convention, not a lock: a plain-text file at the site root that well-behaved crawlers read before fetching anything else. Google, Bing and the major AI crawlers honour it; scrapers and bad actors ignore it, and since the file is public it actually ADVERTISES the paths you list — never use it to hide private pages, use authentication. The classic mistakes it prevents: a Disallow: / that quietly removes a whole site from search, paths without the trailing slash that block more than intended, and forgetting the Sitemap: line that helps discovery most. The AI-crawler section names the published user agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others) so the choice is explicit either way — saying nothing allows them, which is the default everywhere. Blocking Google-Extended affects AI training, not search ranking. Verify the live file at yoursite.com/robots.txt, generate the sitemap it points to with the sitemap generator, and set page-level directives with the meta tag generator.
Frequently asked questions
Does blocking a path in robots.txt hide it?
No — it asks polite crawlers not to fetch it, and the file publicly lists the path. Anything private needs authentication, not robots.txt.
Should I block AI crawlers?
It's a choice, not a rule. Blocking stops the listed bots from using your pages for training or answers; allowing (or saying nothing) lets AI assistants read and cite you. Both are honest options.
Does Crawl-delay work?
Bing and some others honour it; Google ignores it (use Search Console instead). Leave it empty unless a specific crawler is hammering you.
Where does the file go?
At the root: https://yoursite.com/robots.txt, exactly that name. Subdirectories are ignored.