Publish and files

How to write a robots.txt

You need allow, disallow, and a sitemap line. Write them on Robots.txt Generator.

robots.txt is a request to crawlers that honour it, not a lock on the server.
robots.txt is a request to crawlers that honour it, and it is not a lock on the server.

What to run

Open Robots.txt Generator. It stays in the browser.

How to read the result

The file is text for https://example.com/robots.txt.

Disallow: / blocks crawlers that honour it.

It is not access control.

A sitemap line does not create the sitemap.

What to open next

Open HTTP Headers or Is This Website Down on /robots.txt after you upload. Both checks run from Cloudflare.

Open htaccess Redirect Generator if the file must live at the root via a redirect. Usually it should not. The generator stays in the browser.