Where does robots.txt go?

It must be at the root of your domain — https://example.com/robots.txt. Subdirectories are not valid locations.

Does Disallow remove a page from Google?

No. Disallow stops crawling but a blocked URL can still be indexed without content. To remove a page, use a noindex tag or removal request and allow crawling.

Should I add my sitemap to robots.txt?

Yes — a Sitemap: line helps search engines discover your sitemap even before it is submitted in Search Console.

Free · robots.txt

Robots.txt Generator

Build a correct robots.txt with crawl rules and your sitemap. Wrong directives can deindex a site — this keeps it valid.

Preset

User-agent

Disallow (one path per line)

Allow (one path per line)

Crawl-delay (optional)

Sitemap URL

robots.txt

User-agent: *
Disallow:

Upload as /robots.txt at your domain root.

A single wrong line can deindex a site

JW Digital audits crawl rules, indexation and technical SEO so your pages are found — not accidentally blocked.

Technical SEO XML Sitemap Generator →

More free SEO tools

XML Sitemap Generator →

Generate the sitemap.xml you reference in robots.txt.

IndexNow Submitter →

Push URLs to search engines for faster indexing.

Schema Generator →

Add JSON-LD structured data for rich results.