Robots.txt Generator

What this tool does

The Robots.txt Generator builds a standards-compliant robots.txt file. Choose whether to allow all crawlers, add specific disallow paths, configure custom user agents, and include your sitemap URL.

robots.txt controls which crawlers may access which parts of your site. Get it wrong and you can accidentally block your entire site from search engines — a surprisingly common mistake. This tool adds warnings for the exact scenarios that cause that.

The output is plain text you can upload to your site root. It follows the Robots Exclusion Protocol and warns you about invalid paths and sitemap URLs.

How to use it

1. Choose whether to allow all crawlers or block the site.
2. Add any paths to disallow (e.g. /admin, /draft).
3. Optionally add specific user agents like Googlebot or Bingbot.
4. Add your sitemap URL, then copy the generated file to your site root.

How it works

• Each user agent gets its own directive block. "User-agent: *" applies to all crawlers, and specific agents override the wildcard.
• "Allow: /" plus "Disallow: /path" permits crawling except for the listed paths. "Disallow: /" blocks everything.
• The Sitemap directive is appended once, and paths must start with "/" or they are skipped with a warning.

Example

Allow all crawlers but block /admin and /draft, and reference https://example.com/sitemap.xml, produces a clean file with a wildcard block, two disallow rules, and a sitemap line.

Common questions