Robots.txt Generator
Build a robots.txt file that tells search engines and crawlers which parts of your site to skip.
What robots.txt actually controls
A robots.txt file sits at the root of your site and gives crawlers instructions on where they may and may not go. It is how you keep bots out of admin areas, search-results pages, staging directories and other corners that should not be crawled, and it is where you point them to your sitemap. This builds a correctly formatted file from your choices, so you avoid the syntax mistakes that quietly break it.
Pick which paths to disallow and which crawlers to address, and the file is generated ready to upload.
A common and costly misunderstanding
It is worth being clear about what robots.txt does not do: disallowing a page stops well-behaved crawlers fetching it, but it does not make the page private or guarantee it stays out of search results — a blocked URL can still be listed if other pages link to it. For genuinely keeping a page out of the index, a noindex tag is the right tool, and for keeping something truly private, proper access control. Robots.txt manages crawling, not secrecy.
Frequently asked questions
Does robots.txt keep a page out of Google?
Not reliably. It stops crawling, but a blocked page can still appear in results if it is linked elsewhere. Use a noindex tag to keep a page out of the index.
Where does the file go?
At the root of your domain, so it is reachable at yoursite.com/robots.txt.
Is anything uploaded?
No. The file is generated in your browser.