Wildcard group
Parse the rules that apply to User-agent: * rather than confusing them with named bots.
Free robots.txt block checker / wildcard group
Check whether robots.txt blocks an entire website.
Enter any public page to test its site’s robots.txt for the dangerous wildcard Disallow: / pattern, while also reviewing page-level robots directives and the final HTTP response.
✓ No signup✓ Live public response✓ Read-only
Liverobots.txt request
Wildcardcrawler group checked
Exactblanket block verdict
$0no signup required
Answer the dangerous question without overclaiming the rest.
Robots.txt syntax can express many crawler-specific rules. RankLatch’s focused checker verifies the site-wide wildcard block most likely to turn a forgotten staging setting into a production incident.
Blanket disallow
Detect Disallow: / and Disallow: /* as full-site blocking patterns.
Allow exception
Avoid calling the file a blanket block when the wildcard group explicitly allows a path.
Page directives
Show robots meta and X-Robots-Tag values separately from robots.txt behavior.
What the result does—and does not—prove.
- ✓The public robots.txt returned for the final origin
- ✓A wildcard Disallow: / or Disallow: /* rule
- ✓Whether the wildcard group contains an allow exception
- ✓The submitted page’s public robots directives
- —Every named search crawler’s path matching
- —Google’s crawl history or coverage report
- —Rules generated only through private infrastructure
- —A crawl of every URL affected by the file
Current check → continuous protection
A production robots.txt file can change after launch.
RankLatch stores the approved site-wide block state and checks it daily. A newly introduced wildcard block becomes critical evidence instead of a mystery discovered after traffic falls.
See robots.txt monitoring →Straight answers
Before you check.
One live request can verify important public signals. It cannot replace Search Console or prove how a search engine will ultimately treat the page.
01What robots.txt problem does this checker detect?+
It focuses on the high-impact site-wide case: a wildcard user-agent group containing Disallow: / or Disallow: /* without an allow exception. The result also shows this page’s public robots directives.
02Is this a full robots.txt testing engine?+
No. It does not simulate path matching for every named crawler and URL pattern. It deliberately answers whether the public site exposes the blanket wildcard block RankLatch monitors continuously.
03Does a robots.txt block remove pages from Google?+
Robots.txt controls crawling, not index removal by itself. Blocked URLs can sometimes remain indexed without content. This checker does not label Disallow and noindex as the same signal.
04Can an Allow rule change the result?+
Yes. RankLatch does not flag a blanket block when the same wildcard group contains a path-level allow exception, because the file is no longer an unconditional site-wide block.
05Where does the checker request robots.txt?+
It requests robots.txt from the origin of the final public page URL, then evaluates the wildcard group in the returned text.
06What happens if robots.txt later changes?+
Continuous monitoring compares whether the site-wide block state changed and treats a newly introduced blanket wildcard block as critical drift.