Does robots.txt remove a page from Google?

No. robots.txt controls crawling, not indexing. A blocked URL can still be indexed from links, usually without a description. To keep a page out of results, allow crawling and add a noindex rule, or return 404 or 410.

Where does robots.txt go?

At the root of the host, as https://example.com/robots.txt. It only applies to that protocol, host and port, so a subdomain such as shop.example.com needs its own file.

Can I block AI crawlers with robots.txt?

You can ask them not to crawl. Operators such as OpenAI, Anthropic, Google, Apple and Common Crawl publish tokens that honor robots.txt, but some user-triggered fetchers may not, and robots.txt is a request, not access control.

Is anything I paste uploaded?

No. Every tool runs in your browser, so the text, HTML, URLs and images you use are processed on your device and are not sent to our server or stored. Reloading or closing the page clears them.

Yes. Every tool is free with no sign-up and no limits on use. Clearly labeled ads keep it that way.

Will these tools improve my rankings?

They help you get the details right: tags that are complete and valid, previews that read well, files crawlers can parse. No tool can guarantee rankings; those depend on your content, your site and how search engines evaluate them.

Robots.txt Generator

Build a robots.txt file with crawler rules and your sitemap.

100% Free
No Sign Up
Works in Your Browser

Robots.txt generator

Create a robots.txt file, including AI crawler rules

Start from a preset

Group 1

User-agents (comma-separated)

Disallow (one path per line)

Empty = allow everything.

Allow (one path per line)

Exceptions to Disallow.

Group 2

User-agents (comma-separated)

Disallow (one path per line)

Empty = allow everything.

Allow (one path per line)

Exceptions to Disallow.

Sitemaps (one URL per line)

Optional, but helps crawlers find your sitemap.

robots.txt

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /search

# AI training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: CCBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Upload it to the root of your site: https://www.example.com/robots.txt

robots.txt controls crawling, not indexing: a blocked page can still appear in results if other pages link to it. To keep a page out of search, allow crawling and use a noindex meta tag (or remove the page). Blocking AI training crawlers doesn't affect Google Search; Google-Extended only controls use for Gemini.

Everything runs in your browser. Nothing is uploaded.

More SEO Tools

View all tools

Every everyday SEO tool, right in your browser.

No installs. No uploads. No signup.

Browse all 26 tools

SEO tools: 26
Reference guides: 7
Bytes uploaded: 0

How to use the robots.txt generator

Three steps, all in your browser.

1
Start from your page
Have the page's title, URL and key details to hand.
2
Set the rules
Add rules or URLs. The result updates instantly.
3
Copy the result
Copy what you need in one click. Nothing is saved or sent anywhere.

Robots.txt Generator: what to know

robots.txt is a plain-text file at the root of a site that tells crawlers which paths they may crawl. The robots.txt generator builds one from a form: groups of rules for all crawlers or for specific user agents, Allow and Disallow paths, ready-made groups for documented AI crawlers, and your sitemap URL. Copy the result or download it as robots.txt.

The format is standardized in RFC 9309. Each group starts with one or more User-agent lines and lists Disallow and Allow rules; a crawler obeys the group that names it most specifically and ignores the rest. Paths are case-sensitive and match by prefix, with * matching any characters and $ marking the end of a URL. An empty Disallow, or no file at all, means everything may be crawled.

robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it, and Google can't see a noindex tag on a page it isn't allowed to crawl. Use robots.txt to keep crawlers out of infinite spaces such as faceted search or internal search results, and use noindex to keep a page out of results.

Popular uses

Creating robots.txt for a new site
Blocking internal search and filter URLs from crawling
Opting out of documented AI training crawlers
Adding the sitemap location for crawlers to find

Tips for the best result

Never disallow CSS or JavaScript files that pages need to render.
Test the file with the robots.txt tester before you upload it.
Keep a staging site's Disallow: / out of production when you launch.

Read how JustYourSEO works and our privacy details. Text, HTML and images you use here are processed in your browser and are never sent to our server.