Robots.txt Generator

120+ ratingsFree

Build a correct robots.txt with presets and live preview.

Sitemap URL (optional)

Crawl-delay in seconds (optional)

robots.txt

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

ToolHub Calculation Report

🤖 Place this file at your domain root (example.com/robots.txt). Google ignores Crawl-delay; Bing and Yandex honor it. Blocking AI crawlers (GPTBot, ClaudeBot, CCBot) is honored by their operators.

Where the file goes

Robots.txt must live at the root of your domain — https://example.com/robots.txt. Search engines never look anywhere else, and subdomains need their own file.

Presets explained

Standard site allows everything except a common admin path. Block everything is useful for staging sites. Block AI crawlers adds directives honored by GPTBot, ClaudeBot, and CCBot operators.

Crawl-delay caveats

Google explicitly ignores Crawl-delay; Bing and Yandex honor it. If Googlebot hammers your server, use the crawl-rate settings in Search Console instead.

robots.txt is advisory

Well-behaved crawlers obey it, but it is not authentication. Anything truly private must sit behind access control — disallow rules just ask politely.

About the Robots.txt Generator

The Robots.txt Generator creates robots.txt generator on demand, right in your browser. Set the options you need, click generate, and the result is ready to copy. Because everything runs locally, nothing you enter or produce leaves your device.

Generators like this are useful when you need a specific output (a password, a UUID, a QR code, placeholder text) and don't want to install an app or trust an unknown website with your data. This tool is free, has no usage limits, and works the same on phone and desktop.

Common uses: people reach for this tool when they need to use a robots txt generator online, allow googlebot block others, robots txt disallow everything, or sitemap location robots txt.

How to Use This Tool

Set the options the generator exposes (length, format, count, etc.).
Click the generate button to produce the output.
Review the result and adjust the options to regenerate if needed.
Use the copy button to copy the output to your clipboard.

Why Use an Online Generator?

Browser-based tools like this one have a few real advantages over installed software or manual methods:

No installation. It opens instantly in any browser, on any operating system.
Private by default. Everything runs locally, so your data stays on your device.
Always up to date.There's nothing to update — you always get the latest version when you load the page.
Free and unlimited. Use it as often as you like, with no account and no caps.

Frequently Asked Questions

Where do I put robots.txt?

In your domain root, served as plain text at example.com/robots.txt — crawlers only look there. A missing file means everything is allowed; syntax errors can make well-meaning rules fail silently, so keep one rule set per user-agent block.

Does the block-AI preset really work?

It emits explicit disallow rules for GPTBot, ClaudeBot, and CCBot — the crawlers behind OpenAI, Anthropic, and Common Crawl training sets. Those operators currently honor robots.txt, so it works as long as they keep doing so; treat it as a respected request, not a technical wall.

Why is my crawl-delay ignored?

Google has never supported Crawl-delay and skips the line entirely; Bing and Yandex honor it. The generator includes the field for those engines — for Googlebot load problems, use Search Console crawl-rate settings instead.

If Allow and Disallow both match, which wins?

In Google's implementation the more specific (longest) matching path wins. The presets emit conventional, unambiguous combinations — block-everything is a bare Disallow: / with no competing Allow — so you don't have to reason about precedence.

More tools you might find useful

Developer Tools

JSON Formatter

Format and validate JSON with proper indentation.

Developer Tools

JWT Decoder

Decode JWT tokens and inspect header, payload, and signature locally.

Developer Tools

ASCII Table

Every ASCII character with dec/hex/oct/bin codes, searchable.

Developer Tools

Base Converter (Any Base 2-36)

Any base 2-36, fractions included, all common bases at once.

Loading…

About the Robots.txt Generator

Common uses: people reach for this tool when they need to use a robots txt generator online, allow googlebot block others, robots txt disallow everything, or sitemap location robots txt.

Why Use an Online Generator?

Browser-based tools like this one have a few real advantages over installed software or manual methods:

No installation. It opens instantly in any browser, on any operating system.

Private by default. Everything runs locally, so your data stays on your device.

Always up to date.There's nothing to update — you always get the latest version when you load the page.

Free and unlimited. Use it as often as you like, with no account and no caps.

Frequently Asked Questions

Where do I put robots.txt?

Does the block-AI preset really work?

Why is my crawl-delay ignored?

If Allow and Disallow both match, which wins?