How the tester decides
It follows Google’s published robots.txt rules:
- Pick the group. The crawler uses only the most specific “User-agent” group that matches its name (e.g. “Googlebot” beats “*”). Groups for the same name are merged. If no group matches, everything is allowed.
- Find matching rules. Each Allow and Disallow path is compared with the start of the URL path (including any query string).
*matches any characters and$means “end of URL”. - Longest match wins. The rule with the longest path decides. If an Allow and Disallow rule are equally long, Allow wins.
- Paths are case-sensitive.
/Blogdoes not match/blog. Empty Disallow lines block nothing.
Crawler-specific tokens such as Googlebot-Image fall back to the Googlebot group if they do not have their own, as Google describes.
File checks
- A blanket
Disallow: /for all crawlers or Googlebot (common after a staging site goes live). - Rules before any User-agent line, which crawlers ignore.
- Unknown directives and typos (e.g. “Dissallow”).
- “Noindex” lines, which Google stopped supporting in 2019.
- Crawl-delay (ignored by Google) and missing Sitemap lines.
- Files larger than Google’s 500 KiB limit.
Blocking is not the same as de-indexing
A URL blocked in robots.txt can still appear in search results if other pages link to it; Google just cannot see its content. To keep a page out of results, allow crawling and use a noindex meta tag instead. Equally, blocking CSS or JavaScript files can stop Google rendering pages properly.
If pages are allowed but not indexed, work through the Google Indexing Issue Diagnoser, and read what “crawled, currently not indexed” means.