Robots.txt Tester

The robots.txt tester checks whether a URL may be crawled under a robots.txt file. Paste the file, enter a URL path and a user agent such as Googlebot, and it shows whether the URL is allowed or blocked and which rule decided it. It follows the matching rules in RFC 9309, the Robots Exclusion Protocol standard that Google helped write.

Two rules decide most cases. First, a crawler uses only the group whose User-agent matches it most specifically, compared case-insensitively, and falls back to the * group only when no group names it. Second, within that group the rule with the longest matching path wins, and when an Allow and a Disallow match with the same length, Allow wins. Wildcards count toward length, * matches any sequence of characters and $ anchors the end of the URL.

Testing matters because small mistakes have big effects: Disallow: / blocks a whole site, a missing slash blocks more than you meant, and paths are case-sensitive. The tester works on the text you paste, so you can try changes before uploading them. Remember that robots.txt only stops compliant crawlers from fetching; it doesn't remove indexed pages.