Wildcard group parsed
Evaluate the rules that apply to all crawlers rather than treating any disallow line as a sitewide block.
Robots.txt monitoring / sitewide blocks
Catch a sitewide robots.txt block on the next daily check.
A staging rule or deployment can expose Disallow: / on a live domain. RankLatch watches the wildcard crawl state, treats a new blanket block as critical, and records the exact before-and-after evidence without pretending every robots.txt edit is an emergency.
✓ Read-only✓ No plugin✓ No card to try
Dailyrobots.txt check
Criticalnew blanket block
Exactcrawl-state evidence
Read-onlypublic request
Deliberately narrow detection
Interrupt the team for the robots.txt change that can affect everything.
Robots.txt files can contain many legitimate rules. RankLatch does not turn every line edit into noise. It detects the specific transition from crawl allowed to a wildcard sitewide block and preserves that state alongside the monitored pages.
Blanket block detected
Recognize Disallow: / and Disallow: /* as the high-impact crawl state being protected.
Allow exception respected
Avoid calling the state a total block when the wildcard group includes an applicable allow exception.
Critical transition
Elevate a newly introduced blanket block instead of burying it as an ordinary file change.
Exact state evidence
Record Crawl allowed before and Disallow: / after for a clear incident trail.
No line-diff noise
Stay honest about scope: unrelated crawler rules and sitemap edits are not presented as critical RankLatch events.
Evidence over noise
Not every robots.txt edit deserves the same alert.
A dedicated robots.txt versioning tool is better when a team needs every directive and sitemap line diff. RankLatch is for agencies that need a sitewide crawl block surfaced beside the other protected SEO fields.
See the exact methodology →QuestionGeneric page monitorRankLatch
ScopeEvery file editWildcard blanket block
EvidenceFull text diffCrawl allowed → blocked
SeverityConfigured manuallyNew block is critical
Other SEO fieldsUsually separateSame monitoring workspace
How it works
From approved state to useful alert.
RankLatch keeps the workflow deliberately small. Your team stays in control of every site and receives evidence only when a protected field moves.
Read the public file
RankLatch requests the conventional robots.txt location during the monitored site’s scheduled run.
Evaluate the wildcard state
The parser checks for a blanket disallow and accounts for an allow exception before classifying the site.
Alert on the transition
When crawl access changes to a sitewide block, the event records the exact state and restoration step.
Straight answers
Before you monitor.
These limits are intentional. RankLatch protects specific public SEO signals; it does not pretend to explain every ranking movement.
01What robots.txt change does RankLatch monitor?+
RankLatch checks whether the wildcard user-agent group introduces a blanket Disallow: / or Disallow: /* without an allow exception. It compares that sitewide crawl state during each daily run.
02Does RankLatch alert on every robots.txt edit?+
No. This is intentionally not a full line-by-line robots.txt version tracker. RankLatch focuses its critical robots.txt alert on a newly introduced sitewide wildcard block and stays quiet about unrelated rule edits.
03How is a sitewide robots.txt block reported?+
The event identifies the change from crawl allowed to Disallow: /, marks it critical, and provides a direct restoration instruction. Critical monitoring events are eligible for email delivery.
04Does a robots.txt block remove pages from Google immediately?+
Not necessarily. Robots.txt controls crawling rather than directly declaring noindex. RankLatch reports the verified public block without claiming that search engines have already removed pages.
05Will RankLatch bypass a blocked crawler?+
No. RankLatch respects access controls and never attempts to evade bot protection. A site operator can also block the RankLatchBot user agent explicitly.
06How is this different from the noindex monitor?+
A robots.txt blanket block affects crawler access at the site level. A noindex directive appears on an individual response and asks search engines not to index that page. RankLatch monitors both as separate fields.
Protect the approved work
Do not discover a live staging block in next month’s report.
Run the free homepage check, trial up to three sites, then keep one site’s crawl state and key pages protected on Free.
14-day full trial · Free after · no card