Free SEO tool

robots.txt Tester

Test whether a URL is allowed or blocked for any user-agent. This robots.txt tester reads the live robots.txt of the URL's site, or content you paste, with RFC 9309 matching, then shows the deciding group, the deciding Allow or Disallow line with its number, every group found and the declared sitemaps.

How to Test a robots.txt File

Choose where the robots.txt comes from, enter the URL and the user-agent, and submit: the verdict and the line that decided it appear below the form.

Choose "Live robots.txt of the URL's site" to read the file from the URL's site, or "Pasted content" to test a draft.
Enter the full URL you want to test, including https://.
Enter the user-agent, such as Googlebot or Bingbot. A full user-agent string also works, and * tests the group for all crawlers.
If you chose "Pasted content", paste the robots.txt text into the box.
Submit the form and read the verdict, the deciding group and the deciding line.

robots.txt Disallow and Allow Results Explained

The result says whether the user-agent may crawl the URL and names the group and the line that decided it.

Allowed or Blocked

Allowed means no rule of the matching group forbids the URL's path for the user-agent you entered. Blocked means the deciding line is a Disallow rule, or the live file was unreachable. A blocked URL is not crawled, but it can still be indexed when other pages link to it.

robots.txt User Agent and the Deciding Group

The tester reduces what you type to a product token, such as googlebot, and compares it without regard to case. If a group names that token, its rules apply, and groups naming the same token are merged; otherwise the * group applies. When neither exists, every URL is allowed for that user-agent.

The Deciding Line

The deciding line is the Allow or Disallow rule that settled the verdict, shown with its line number in the file. The rules table lists every rule of the matching group and marks which ones match the path. When no rule matches, the URL is allowed, and /robots.txt itself is always allowed.

robots.txt Allow and Disallow Precedence

When several rules match, the one with the longest pattern wins, as RFC 9309 specifies. If an Allow and a Disallow pattern have the same length, Allow wins. For example, Allow: /private/public-page beats Disallow: /private/ for that page.

robots.txt Wildcard Patterns

An asterisk matches any sequence of characters, and a dollar sign at the end anchors the pattern to the end of the URL. Disallow: /*.pdf$ blocks /files/a.pdf but not /files/a.pdf?x=1, because the query string is part of what rules match. Percent-encoded characters are normalized before the comparison.

When robots.txt Is Missing or Unreachable

A 4xx answer means the file is unavailable, and RFC 9309 lets crawlers access every URL, so the tester reports Allowed. A 5xx answer means it is unreachable, and crawlers must assume every URL is disallowed, so the tester reports Blocked. A timeout or a refused connection gives no verdict.

Live robots.txt or Pasted Content

Live mode tests the file crawlers see now; pasted mode tests a draft without requesting anything from the site.

How to Check the robots.txt of a Website

In live mode the tester requests /robots.txt once from the site of the URL you enter and follows up to five redirects. It parses the first 500 KiB, the minimum RFC 9309 asks crawlers to read, and links the file it read in the result.

How to Test Rules Before You Publish Them

Pasted mode parses the text you paste, up to 500 KiB, with the same parser as live mode. Only the path and query of the URL are used, so you can compare the current file and your fix on the same URL before you upload it.

robots.txt Tester FAQ

How Does robots.txt Work?

A robots.txt file at the root of a site lists groups of rules, each starting with one or more User-agent lines. A crawler follows the group that names it, or the * group, and checks the URL path against its Allow and Disallow rules. The file is a request that well-behaved crawlers honour, not access control.

Is robots.txt Case Sensitive?

User-agent names and field names such as Disallow are matched without regard to case. Paths are case sensitive, so Disallow: /Private does not block /private. The file itself must be named robots.txt, in lower case, at the root of the host.

Why Is My robots.txt Not Working?

A robots.txt file applies only to the scheme, host and port it is served from, so a file on www.example.com does not cover example.com. Check that a group names the crawler's token, that the pattern starts with / and matches the case of the URL, and that a longer Allow rule is not overriding your Disallow. Test the live file, then your pasted fix, to confirm the change.

Does Blocking a URL in robots.txt Remove It From Google?

No. robots.txt controls crawling, not indexing, and Google can still index a blocked URL that other pages link to, usually without a description. To keep a page out of search results, let it be crawled and use a noindex meta tag or X-Robots-Tag header.

Can I Test a robots.txt File Before Publishing It?

Yes. Choose "Pasted content", paste the draft and enter the URL and user-agent you want to check. Nothing is requested from the site in this mode, and the verdict comes from the same parser as a live test.