Robots.txt Generator by Federico Calò
Generate a robots.txt file with pre-set presets or customized rules for the user-agent. Block AI bots, unwanted crawlers and add Sitemap and Crawl-delay directives.
Configure
Keep it brief
No generated output
Fill out the form and click the button to generate output.
What is robots.txt and what does it serve?
The robots.txt file is a text file in the site's root that instructs crawlers (bots) on which pages to or not visit. It's used to exclude private areas, reduce server load and monitor indexing.
Are you blocking only AI bots like GPTBot and CCBot?
Use the preset "Block common AI bots" to generate Disallow rules for GPTBot, ChatGPT-User, CCBot, anthropic-ai and Google-Extended, while keeping access open to regular search crawlers like Googlebot.
Does the robots.txt file guarantee the privacy of excluded pages?
Don't send robots.txt - it's a voluntary convention: respectful bots follow it, but malicious bots or scrapers don't obey. To protect sensitive content use authentication, passwords, or server-side access restrictions.
Where should I place the robots.txt file?
File must be exactly at the root of the domain: https://example.com/robots.txt. Does not work in subfolders. If you manage multiple subdomains (blog.example.com), each subdomain must have its own robots.txt file.