Skip to content

Robots.txt Tester

Test whether a crawler may fetch a URL under your robots.txt.

How to use the robots.txt tester

Three steps, all in your browser.

  1. 1

    Paste robots.txt

    Paste the file, then enter a URL and a user agent to test.

  2. 2

    Set the rules

    Add rules or URLs. The result updates instantly.

  3. 3

    Copy the result

    Copy what you need in one click. Nothing is saved or sent anywhere.

Robots.txt Tester: what to know

The robots.txt tester checks whether a URL may be crawled under a robots.txt file. Paste the file, enter a URL path and a user agent such as Googlebot, and it shows whether the URL is allowed or blocked and which rule decided it. It follows the matching rules in RFC 9309, the Robots Exclusion Protocol standard that Google helped write.

Two rules decide most cases. First, a crawler uses only the group whose User-agent matches it most specifically, compared case-insensitively, and falls back to the * group only when no group names it. Second, within that group the rule with the longest matching path wins, and when an Allow and a Disallow match with the same length, Allow wins. Wildcards count toward length, * matches any sequence of characters and $ anchors the end of the URL.

Testing matters because small mistakes have big effects: Disallow: / blocks a whole site, a missing slash blocks more than you meant, and paths are case-sensitive. The tester works on the text you paste, so you can try changes before uploading them. Remember that robots.txt only stops compliant crawlers from fetching; it doesn't remove indexed pages.

Popular uses

  • Checking a URL isn't blocked before a launch
  • Debugging why a page isn't crawled
  • Testing new rules before uploading them
  • Verifying what an AI crawler is allowed to fetch

Tips for the best result

  • Test real URLs from your site, including query strings.
  • Test the same URL for Googlebot and * to catch group mix-ups.
  • Remember Google caches robots.txt for up to 24 hours, so changes aren't instant.

Learn more

Read how JustYourSEO works and our privacy details. Text, HTML and images you use here are processed in your browser and are never sent to our server.

Frequently asked questions

Under RFC 9309, the rule with the longest matching path wins. If an Allow and a Disallow match with the same length, Allow wins. So Allow: /shop/sale beats Disallow: /shop for /shop/sale/shoes.

The group whose User-agent line matches the crawler's product token, compared case-insensitively. A crawler follows only its most specific group and ignores the * group, so rules under * don't add to a named group.

Yes. Disallow: /Private doesn't block /private. User-agent names are case-insensitive, but paths are matched exactly, with * matching any characters and $ marking the end of the URL.

No. The text, HTML, files and images you paste or open are processed on your device and are not sent to our server or stored. The five URL tools (SEO audit, indexability checker, redirect checker, HTTP header checker and broken link checker) are the exception: they send the URL you enter to our server, which requests that public page (and, for the broken link checker, each link on it) and returns what it received to your browser to check. The URL and the page aren't stored or logged. The optional Import from URL in the meta tag, Open Graph and Twitter card generators works the same way.

Yes. Every tool is free with no sign-up and no limits on use. Clearly labeled ads keep it that way.

They help you get the details right: tags that are complete and valid, previews that read well, files crawlers can parse. No tool can guarantee rankings; those depend on your content, your site and how search engines evaluate them.