Robots.txt Generator

Create a robots.txt file to control how search engines crawl your site.

Default policy

Disallow paths (one per line, e.g. /admin/)

Sitemap URL (optional)

Generated robots.txt

How to use

Choose whether to allow or block crawlers by default.
Add any paths to disallow and your sitemap URL.
Copy the generated robots.txt to your site root.

Frequently asked questions

What is robots.txt?

A plain-text file at your site root that tells search-engine crawlers which parts of your site they may or may not crawl.

Where does robots.txt go?

At the very root of your domain, for example example.com/robots.txt. It must be reachable there to work.

🔒 This tool runs entirely in your browser. Nothing you enter is uploaded, logged, or stored.

Related tools