Skip to content

Robots.txt Generator

Build a robots.txt file with crawler rules and your sitemap.

How to use the robots.txt generator

Three steps, all in your browser.

  1. 1

    Start from your page

    Have the page's title, URL and key details to hand.

  2. 2

    Set the rules

    Add rules or URLs. The result updates instantly.

  3. 3

    Copy the result

    Copy what you need in one click. Nothing is saved or sent anywhere.

Robots.txt Generator: what to know

robots.txt is a plain-text file at the root of a site that tells crawlers which paths they may crawl. The robots.txt generator builds one from a form: groups of rules for all crawlers or for specific user agents, Allow and Disallow paths, ready-made groups for documented AI crawlers, and your sitemap URL. Copy the result or download it as robots.txt.

The format is standardized in RFC 9309. Each group starts with one or more User-agent lines and lists Disallow and Allow rules; a crawler obeys the group that names it most specifically and ignores the rest. Paths are case-sensitive and match by prefix, with * matching any characters and $ marking the end of a URL. An empty Disallow, or no file at all, means everything may be crawled.

robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it, and Google can't see a noindex tag on a page it isn't allowed to crawl. Use robots.txt to keep crawlers out of infinite spaces such as faceted search or internal search results, and use noindex to keep a page out of results.

Popular uses

  • Creating robots.txt for a new site
  • Blocking internal search and filter URLs from crawling
  • Opting out of documented AI training crawlers
  • Adding the sitemap location for crawlers to find

Tips for the best result

  • Never disallow CSS or JavaScript files that pages need to render.
  • Test the file with the robots.txt tester before you upload it.
  • Keep a staging site's Disallow: / out of production when you launch.

Learn more

Read how JustYourSEO works and our privacy details. Text, HTML and images you use here are processed in your browser and are never sent to our server.

Frequently asked questions

No. robots.txt controls crawling, not indexing. A blocked URL can still be indexed from links, usually without a description. To keep a page out of results, allow crawling and add a noindex rule, or return 404 or 410.

At the root of the host, as https://example.com/robots.txt. It only applies to that protocol, host and port, so a subdomain such as shop.example.com needs its own file.

You can ask them not to crawl. Operators such as OpenAI, Anthropic, Google, Apple and Common Crawl publish tokens that honor robots.txt, but some user-triggered fetchers may not, and robots.txt is a request, not access control.

No. The text, HTML, files and images you paste or open are processed on your device and are not sent to our server or stored. The five URL tools (SEO audit, indexability checker, redirect checker, HTTP header checker and broken link checker) are the exception: they send the URL you enter to our server, which requests that public page (and, for the broken link checker, each link on it) and returns what it received to your browser to check. The URL and the page aren't stored or logged. The optional Import from URL in the meta tag, Open Graph and Twitter card generators works the same way.

Yes. Every tool is free with no sign-up and no limits on use. Clearly labeled ads keep it that way.

They help you get the details right: tags that are complete and valid, previews that read well, files crawlers can parse. No tool can guarantee rankings; those depend on your content, your site and how search engines evaluate them.