Robots.txt Generator Online

A robots.txt file sits at the root of your website and tells search engine crawlers which pages they're allowed to index. Getting it wrong — blocking crawlers from your whole site or from pages you want indexed — is a surprisingly common SEO mistake. This tool generates a correct, standards-compliant robots.txt for you.

How it works

You select which bots to configure (all crawlers, Googlebot, Bingbot, or others by name), specify which paths to allow or disallow, and set your sitemap URL. The tool assembles the correct syntax and lets you copy or download the result. Everything runs client-side.

How to use it

1. Choose the User-agent — use * to apply rules to all crawlers, or enter a specific bot name
2. Add Disallow paths for pages you don't want indexed (e.g. /admin/, /checkout/)
3. Add any explicit Allow paths if needed (useful to allow a specific path within a disallowed directory)
4. Enter your Sitemap URL (e.g. https://yourdomain.com/sitemap.xml)
5. Copy the generated file and place it at https://yourdomain.com/robots.txt

A few things worth knowing

• robots.txt is advisory, not enforced: Well-behaved crawlers (Googlebot, Bingbot) respect it. Malicious bots and scrapers don't. Don't use it to hide sensitive content
• Blocking a page doesn't remove it from search results: If a page is already indexed and you then block it in robots.txt, Google may still show it in results because it can't crawl the page to see the noindex tag. Use noindex in the HTML for pages you want out of search results
• Trailing slashes matter: /admin disallows that exact path. /admin/ disallows the directory and everything under it

FAQ

Where do I put the robots.txt file? It must be at the root of your domain: https://yourdomain.com/robots.txt. Placing it in a subdirectory doesn't work.

Should I block my staging site? Yes — use Disallow: / on staging environments to prevent them from being indexed alongside your live site.

Can I have multiple User-agent blocks? Yes. Separate each bot's rules into its own block. Each block starts with a User-agent: line.

Related Tools

• Sitemap Generator
• Meta Tag Generator