T TextTools

Robots.txt Generator

Build a robots.txt — allow / disallow rules per user-agent, optional crawl-delay, and sitemap URL.

Add directives below, then Generate.

Rules (one per line: allow:/path, disallow:/path, or useragent:Name)

Advertisement

How to use Robots.txt Generator

  1. Type rules — one per line: useragent:*, disallow:/admin/, allow:/admin/public/, etc.
  2. Optionally add sitemap URL and crawl-delay.
  3. Click Generate and save as robots.txt at your site root.

What is Robots.txt Generator?

robots.txt tells search-engine crawlers which parts of your site to index and which to skip. This generator takes simple rules in the format useragent:Name, disallow:/path, and allow:/path and outputs a properly-formatted file ready to upload to your site root.

Advertisement

FAQ

Where does robots.txt go?
Always at the root of your domain: https://yoursite.com/robots.txt — not in subdirectories.
Is robots.txt enforceable?
No — it's a suggestion that well-behaved crawlers respect. For real access control, use authentication or server-side blocks.

Related tools