Robots.txt Generator

Robots.txt Generator

Default - All Robots are:	Allowed Refused
Crawl-Delay:	Default - No Delay 5 Seconds 10 Seconds 20 Seconds 60 seconds 120 Seconds
Sitemap: (leave blank if you don't have)
Search Robots:	Google	Same as Default Allowed Refused
	Google Image	Same as Default Allowed Refused
	Google Mobile	Same as Default Allowed Refused
	MSN Search	Same as Default Allowed Refused
	Yahoo	Same as Default Allowed Refused
	Yahoo MM	Same as Default Allowed Refused
	Yahoo Blogs	Same as Default Allowed Refused
	Ask/Teoma	Same as Default Allowed Refused
	GigaBlast	Same as Default Allowed Refused
	DMOZ Checker	Same as Default Allowed Refused
	Nutch	Same as Default Allowed Refused
	Alexa/Wayback	Same as Default Allowed Refused
	Baidu	Same as Default Allowed Refused
	Naver	Same as Default Allowed Refused
	MSN PicSearch	Same as Default Allowed Refused
Restricted Directories:	The path is relative to root and must contain a trailing slash "/"

Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

About Robots.txt Generator

Robots.txt Generator — Create a Properly Formatted File in Seconds

Build a robots.txt file by selecting which bots you want to allow or restrict, which paths to block, and where your sitemap lives — then copy the result straight into your site's root directory.

What a robots.txt file actually does

It tells search engine crawlers which parts of your site they're allowed to request — commonly used to keep bots away from admin areas, internal search result pages, or staging folders you don't want showing up in search. It does not remove a page that's already indexed, and it does not make private content secure — anyone can view a site's robots.txt file directly, so never list genuinely sensitive paths in it.

What it generates

• User-agent rules — allow or disallow specific bots (Googlebot, Bingbot, or all bots)
• Disallow paths — block specific folders or URL patterns from being crawled
• Sitemap reference — point crawlers to your XML sitemap so they can discover your pages more efficiently
• Crawl-delay (where supported) — space out how often a bot hits your server

How to use it

1. Select which bots the rules should apply to
2. Add the paths you want to block (if any)
3. Include your sitemap URL — you can generate one with our XML Sitemap Generator if you don't have one yet
4. Copy the output and upload it as robots.txt to your site's root directory (not a subfolder)

A common mistake to avoid

A misconfigured robots.txt is one of the most common causes of a site's pages failing to get indexed at all — a single Disallow: / left in place from a staging site, for example, blocks your entire site from every search engine. After uploading, verify the file with our Meta Tags Analyzer or Google Search Console's own robots.txt tester before assuming it's working correctly.