In short and in plain language
Sometimes pages of a shop vanish from search engines. Often a single setting is the cause. It may come from a test system into the live shop.
The shop keeps running as normal. That is why the fault is often noticed late.
The article shows how to check this. For each important page, you set how it should be. A check reports every difference.
We set up this check for your shop.
A shop that drops out of the index reports no error. The server answers with 200, the cart works, load times look normal – and yet category and product pages disappear from search results within days. The trigger is usually a single line: a robots directive that travelled from a staging environment into production. Such directives are not a fringe topic but everyday practice – 47.0% of desktop pages and 47.9% of mobile pages serve one (Web Almanac 2025). That is precisely why a flipped directive goes unnoticed for so long. This article describes a five-stage check chain that makes the state measurable before the search console confirms it weeks later.
Key takeaways
- An accidental noindex produces no error status. The page answers with 200, checkout works – and the search engine still drops it entirely from the results, regardless of how many links point to it (Google Search Central).
- Blocking directives are rare: noindex sits on 3.5% of desktop and 2.4% of mobile pages, nofollow on 2.4% and 2.8% (Web Almanac 2025). The second delivery path, the X-Robots-Tag response header, reaches 0.6% and is rarely opened during troubleshooting.
- The check chain has five stages: fetch and status, robots.txt, meta robots, canonical host, sitemap status. Inner pages carry a directive more often at 50% than home pages at 46% (Web Almanac 2025) – a chain that only queries the home page measures the wrong addresses.
- robots.txt is not a tool for keeping a page out of the index: a blocked address can still appear in the results, just without a snippet (Google Search Central). Blocking and setting noindex at the same time prevents the directive from ever being read.
- The search console is a lagging indicator: the example list is capped at 1,000 rows, confirming a fix typically takes up to about two weeks (Google Search Console help), and it can take months before the crawler revisits a page (Google Search Central).
- The scale is quantifiable: German online retail last reported EUR 92.3 billion in turnover (HDE), and enterprises in the EU generate 19.49% of total turnover from e-sales (Eurostat). Of that, 8.39 percentage points come from websites or apps, the rest from EDI.
How a noindex reaches production
The path is almost invariably the same. A staging environment carries the directive for good reason, so its content stays out of the index. Then a clone is pulled into production, an environment variable stays set, a maintenance mode is only half switched off, or a rule on the upstream proxy sets the response header for every request. In staging environments and release paths this behaviour is intended; the fault appears at the handover, not at the moment the directive is set. And because the handover is rarely logged, there is no timestamp a later investigation could start from.
The second group of causes sits in the extensions. An SEO module loses its configuration after an update and falls back to a default; a category filter sets noindex across parameter pages and catches the main category with it; an expired licence switches a module into a reduced mode of operation. The fact that extension license expiry changes the feature set is itself a maintenance topic – and a trigger that appears in no deployment log, because nobody published anything that day.
A status 200 is no proof of visibility
What the corpus figures say about robots directives
Robots directives are the norm, not the exception: 47.0% of desktop pages and 47.9% of mobile pages serve one at all (Web Almanac 2025). Only a small share of those is blocking, and the source reports the two rules separately: noindex sits on 3.5% of desktop pages and 2.4% of mobile pages, nofollow on 2.4% of desktop pages and 2.8% of mobile pages (Web Almanac 2025, figure 5.8). The running text of the same source combines both rules into 2.8% desktop and 2.4% mobile; that combined value falls below the noindex share reported there, so it is not used here as a headline figure. For monitoring, that implies an awkward property: the line sits on almost every second page, its content rarely flips, and so nobody goes looking for it.
Two further figures explain why troubleshooting often starts in the wrong place. The X-Robots-Tag response header reaches 0.6% of desktop pages and 0.7% of mobile pages (Web Almanac 2025) – it is so rare that it escapes notice when someone checks the source code, even though it has the same effect. And 10.1% of desktop pages and 10.3% of mobile pages contain invalid elements in the document head (Web Almanac 2025); a head that ends prematurely can push the meta robots line out of the rendered document even though it sits in the source.
| Stage | What is measured | Corpus value | Typical failure |
|---|---|---|---|
| Fetch and status | HTTP status per target address | all 4xx except 429 are treated alike (Google Search Central) | product page silently drops to 404 after catalogue maintenance |
| robots.txt | Status code, size, content | 85% of fetches return a valid 200 (Web Almanac 2025) | a disallow from the staging environment stays in place |
| Meta robots | Directive in the rendered document | 3.5% carry noindex, 2.4% nofollow (desktop, Web Almanac 2025) | a default value after a module update |
| X-Robots-Tag | Directive in the response header | 0.6% use the response header (Web Almanac 2025) | a rule on the upstream proxy hits every response |
| Canonical host | Target of the canonical annotation | 68% of desktop pages set a canonical (Web Almanac 2025) | the canonical points at the staging host after a migration |
| Sitemap status | Reachability, size, limits | 50,000 addresses per file (sitemaps.org) | the file breaks the limit and is not processed in full |
The table also shows why a single measurement is not enough. Each stage can look clean on its own while the combination makes the page invisible: an address with a clean 200 and a correct meta line has no effect if robots.txt blocks the fetch – because then the meta line is not read at all (Google Search Central). Measuring one stage only yields a green result with the wrong meaning.
The five stages of the check chain
A dependable chain runs every sample through the same five stages and stores the result per stage separately. The selection of addresses matters: not the home page, but a fixed set of categories, product pages, filter pages and content pages. Inner pages carry a robots directive more often at 50% than home pages at 46% (Web Almanac 2025) – querying only the home page misses exactly the addresses that flip. Technically such a chain sits next to the rest of the operational monitoring and differs little from an availability check; only the subject of the check is different.
Stage 1: fetch and status
Status code, redirect chain and response size are recorded per target address. Crawlers follow up to 10 redirect hops by default (Google Search Central); anything beyond that ends without a result.
Stage 2: robots.txt
Status code, file size and content are compared against the last confirmed state. Google reads the file up to 500 KiB and ignores everything after that (Google Search Central).
Stage 3: meta robots
The directive is read in the rendered document, not in the raw source. If two annotations conflict, the more restrictive rule applies (Google Search Central).
Stage 4: canonical host
The check confirms that the canonical annotation points at the same host and the same scheme. 68% of desktop pages now set one (Web Almanac 2025).
Stage 5: sitemap status
Reachability, address count and file size are held against the limits of the protocol: 50,000 addresses and 52,428,800 bytes per file (sitemaps.org).
Extra track: X-Robots-Tag
The response header is evaluated separately because it has the same effect and stays invisible in the source. Only 0.7% of mobile pages use it (Web Almanac 2025).
Every stage needs an expected state that is kept under version control: this address is indexable, that one is blocked on purpose. Without that reference state the chain reports every intentional block as a fault and gets switched off after two weeks, because the on-call rota can no longer separate the noise. The reference state belongs in the same administration as the other maintenance agreements, so changes to it stay traceable instead of surfacing as an accidental finding.
Stages 1 and 2: fetch, status and robots.txt
The fetch is the cheapest stage and finds the coarsest faults. A 4xx on a product page removes it from the index without any warning appearing anywhere: all 4xx except 429 are treated alike and reported to the downstream system as content that does not exist (Google Search Central). That is why every sample needs the status code at the end of the redirect chain, not the first value. How faulty addresses can be tracked down systematically is covered in the article on 404 and redirect maintenance.
For robots.txt the status code is already a statement in itself. 85% of all fetches in the corpus crawl ended with a valid 200 (Web Almanac 2025), 13% with a 404, down from 14% the year before (Web Almanac 2025). Neither is critical. The third case is: if the server answers with a server error, Google stops crawling the site for the first 12 hours and then works with the last good version for up to 30 days (Google Search Central). A fault at this point therefore runs on for a long time before it becomes visible.
# Stage 1: status at the end of the redirect chain
curl -sS -o /dev/null -L -w '%{http_code} %{url_effective} %{num_redirects}\n' "$URL"
# Stage 2: robots.txt with status code and size
curl -sS -o /tmp/robots.txt -w '%{http_code} %{size_download}\n' "$HOST/robots.txt"
grep -n -i -E '^(disallow|allow|sitemap):' /tmp/robots.txt
# Stage 3: read meta robots and X-Robots-Tag separately
curl -sS -L "$URL" | grep -o -i '<meta[^>]*name="robots"[^>]*>'
curl -sS -I -L "$URL" | grep -i '^x-robots-tag:'
# Stage 4: compare the canonical against the expected value
curl -sS -L "$URL" | grep -o -i '<link[^>]*rel="canonical"[^>]*>'
Two limits belong in the evaluation. Google reads robots.txt up to 500 KiB; content beyond that size is ignored (Google Search Central), and the standard requires crawlers to parse at least 500 kibibytes (IETF, RFC 9309). In practice size is rarely the problem: 98% of files stayed below 100 KB (Web Almanac 2025). The silent case is more common – 1.8% of desktop fetches and 1.7% of mobile fetches returned an empty file (Web Almanac 2025), which looks healthy in any superficial visual check. Automated checks do find it: in the previous edition, 8.43% of desktop pages and 7.40% of mobile pages failed the check for a valid robots.txt (Web Almanac 2024).
Latency belongs in the measurement too. A corrected robots.txt does not take effect immediately: Google generally caches the file for up to 24 hours (Google Search Central), and the standard sets the same upper bound (IETF, RFC 9309). If the file sits behind a redirect chain, crawlers should follow at least five consecutive redirects (IETF, RFC 9309) – beyond that, reachability is no longer assured. Sounding the all-clear right after a correction measures the cache, not the effect.
Stage 3: meta robots and the precedence of the strictest rule
The third stage reads the directive where it takes effect. Two delivery paths are equivalent: the meta line in the document and the X-Robots-Tag response header. If they conflict, the more restrictive rule wins (Google Search Central). A forgotten test switch in the response header therefore beats the correct meta line, and because the header stays invisible in the source, troubleshooting looks in the wrong place. That is exactly why the chain reads both paths separately and logs both results side by side.
The consequences are uncomfortably asymmetric. Once the directive has been read, the search engine removes the page entirely from the results, regardless of how many links point to it (Google Search Central). The return takes far longer: it can take months before the crawler revisits a page (Google Search Central). A deployment that removes the line therefore does not restore the previous state – it only stops the further decline.
Blocking and excluding are two different things
Stages 4 and 5: canonical host and sitemap status
Canonical annotations are standard by now: 68% of desktop pages and 67% of mobile pages carry one, up from 65% on both device classes the year before (Web Almanac 2025). That turns the canonical host into a fault source of its own. If the annotation points at the staging host after a migration, or at an address without TLS, the page itself refers to a document that is not meant to be delivered. The check chain therefore compares not only whether a canonical annotation exists, but whether scheme, host and path match the reference state.
The sitemap is the fifth stage and the only one with hard numbers. One file holds 50,000 addresses or 50 MB uncompressed (Google Search Central); the protocol states the same limit down to the byte with 52,428,800 bytes (sitemaps.org). If either is broken, the crawler no longer processes the file in full – and because a shop with a growing catalogue drifts slowly towards that limit, the moment goes unnoticed without measurement. Two carefully maintained fields have no effect at all: priority and changefreq are ignored (Google Search Central).
- Define and version a fixed sample of home page, categories, product pages, filter pages and content pages
- Record status code at the end of the redirect chain, robots.txt state, meta robots, X-Robots-Tag, canonical target and sitemap entry per address
- Store a reference state per address: indexable or deliberately blocked, with a reason and a date
- Run the chain on every deployment and on a schedule as well, not only after larger releases
- Treat deviations as their own alert class, separate from availability and response time notifications
- Keep results for at least a year so the point in time of a change stays determinable later
Why the search console alone arrives too late
The Search Console page indexing report is a good tool and a poor alerting mechanism. Per issue it shows only a capped example list: the list is limited to 1,000 rows and does not necessarily contain every affected address (Google Search Console help). The number in the chart is complete, the list below it is not – deriving the scale of an incident from the list understates it considerably for any larger shop.
Then there is the latency. It takes about a week before a new page is crawled and indexed at all (Google Search Console help). Confirming a reported fix typically takes up to about two weeks, and in individual cases considerably longer (Google Search Console help). And an issue stays the same case until 90 days have passed since the last observed instance (Google Search Console help). For post-incident work that is usable; as an early warning it is too slow.
The removals tool is not a repair either, but a time window. A successful request only lasts about six months (Google Search Console help), and even the fast path usually takes up to a day, while acceptance stays open in the individual case (Google Search Console help). The tool buys time when confidential addresses have ended up in the index; it does not remove the cause. Anyone planning it as an emergency brake should document the procedure together with the rest of the emergency path.
Alerting: thresholds, quiet periods, escalation
A check chain without an alerting path is a log nobody reads. Three levels make sense. A single address with an unexpected noindex is a note and gets collected. Two or more addresses from the same page template are an incident and fire immediately. A deviation on robots.txt or the canonical host affects the entire catalogue on principle and goes straight to the on-call rota without a collection window – the same logic that applies to silent background job failures.
Separating this from the rest of the notification flood matters. Indexability rarely produces alerts, but each one is expensive; availability notifications are frequent and mostly harmless. If both land in the same channel, the on-call rota gets used to the noise and dismisses both equally fast. How channels, thresholds and quiet periods can be separated cleanly is covered in the article on log monitoring and alerting; the check chain gets its own channel there, with its own escalation level.
The output shows what a good notification has to deliver: it names the stage, the affected address, the expected and the actual value, and the time of the last clean measurement. Without that last value every investigation starts with the question of how long the state has existed – and that cannot be answered without a stored history.
What an unnoticed loss of visibility costs
The order of magnitude can be derived from the market. German online retail last grew by 3.9% to EUR 92.3 billion (HDE), and the online share of total retail sits at 13.5%, up from 13.4% the year before (HDE). At company level the share is even clearer: enterprises in the EU generated 19.49% of their total turnover from e-sales (Eurostat), of which 8.39 percentage points came from orders via websites or apps and 11.07 percentage points from EDI-type messages, and the share of enterprises with e-sales rose from 18.93% in 2014 to 23.59% in 2024 (Eurostat).
These figures prove no specific damage in an individual case – they frame what is at stake. A shop whose category pages are unfindable for three weeks loses the share of its turnover that comes from organic search, and does not win it back automatically: recovery depends on the crawl rhythm, not on the deployment. The guidance on crawl budget names 1 million unique pages as the threshold for large sites and 10,000 unique pages for content that changes very rapidly (Google Search Central) – below that, an indexing problem is almost invariably a configuration problem, not a capacity problem.
Availability and findability are two different measurements. Monitoring only the first means learning about the loss of the second from the revenue report.
Building the check chain into the maintenance rhythm
Technically the chain is manageable: one script, one reference state, one history, one alert channel. The effort sits in the upkeep. Every new page template belongs in the sample, every intentional block in the reference state, every change to robots.txt in the history. That is why the chain belongs in the same rhythm as security updates, ongoing shop maintenance and content duties such as the statutory warranty label on product pages – as a fixed item, not as a project.
Two extensions are worth the effort. First, a reconciliation between sitemap and sample: addresses that appear in the sitemap but carry a noindex are a reliable sign of a configuration fault. Second, a link with the operational key figures, so that a drop in visibility appears in the same report as response times and availability – the way it is already customary in performance monitoring within a maintenance contract.
A start that stands within a day
Sources and studies
Related Articles
Disk space monitoring before the shop stops
A full data partition does not cause an outage, it causes stuck orders. Which directories grow, which threshold works and what a dated cleanup cycle covers.
WordPress security updates: what old branches still get
WordPress 7.1.2 closed a critical flaw and the fix was backported to 4.7 as a courtesy. What older branches receive, what they do not, and when to move up.
Discount codes in Black Week: detecting abuse early
Discount codes in Black Week: code length, a rate limit at the coupon field, alerts on failed attempts and new-customer discounts that resist multiple accounts.