Formatters & Code
Robots.txt Generator
Generate a robots.txt that allows or blocks crawler paths.
Results appear here.
What the result means
The text starts with a User-agent line for an asterisk, which addresses every crawler that honours this file. When allow all stays on, the next line is Allow slash. Each non-empty line in the disallow box becomes its own Disallow rule, in the order typed. A sitemap address, if you entered one, is added as a Sitemap line. Copy puts the text on the clipboard, and download saves robots.txt. Put that file at the root of the host you mean. A copy inside a folder is easy for crawlers to miss. The file is a request to automated clients, not a lock on the server. Someone who knows a URL can still open it, and a private page still needs real authentication.
Related tools
Questions
Does this stop humans?
No. A person who knows the URL can still open it. A robots.txt is a note to crawlers, not access control, and it does not hide a page that is linked elsewhere.
Where does the file go?
Save it as robots.txt at the root of the host, so the address looks like https://example.com/robots.txt. A copy in a subfolder is often ignored.
What if disallow is left empty?
With allow all ticked, the file is User-agent star and Allow slash. Disallow lines appear only for non-empty lines in the box. A Sitemap line appears only when you supply an address.