A site that intermittently returns server errors doesn't just lose real user visits for as long as the outage lasts: it changes, in a measurable way with effects that persist after the problem is fixed, how Google crawls that domain. This guide is about that specific technical impact, distinct from the availability a user perceives: what exactly happens when Googlebot hits an outage, how trust gets rebuilt, and what the difference is, at the crawling-algorithm level, between a one-off outage and sustained instability.
What Googlebot does when it hits 5xx errors
When Googlebot gets a 5xx response code (server error) trying to crawl a URL, it doesn't treat that as an isolated failure of that specific page but as a signal about the overall health of the server serving it. If the volume of 5xx errors rises above what's normal for that domain, the crawl-rate management system automatically reduces crawl speed, in effect giving the server "more room" on the assumption that it's overloaded or unstable. It's a defensive response designed not to worsen an existing infrastructure problem, but it carries a cost: less crawling means new or changed content takes longer to be discovered and updated in the index for as long as that reduction lasts.
The special case of a downed robots.txt
There's a specific, well-documented behavior from Google for the robots.txt file itself: if Googlebot gets a server error trying to fetch it, sustained for more than 12 hours, the system starts assuming the entire site is blocked from crawling, as a precaution, until robots.txt starts responding normally again. It's a nuance that surprises many technical teams: the whole site doesn't need to be down to lose crawling entirely, it's enough for that specific robots.txt file to stop responding reliably during that window, even if the rest of the site keeps functioning normally.
Short outage versus sustained instability: not the same impact
In how its system actually reacts, Google distinguishes between a one-off interruption of a few hours and a pattern of instability that repeats over days or weeks. A short, isolated outage, especially if the site has a prior track record of stability, is normally absorbed with no lasting consequences: Google can retry later and the crawl rate recovers quickly once the server starts responding normally again. The real problem is sustained instability: intermittent failures repeating over several days, even if each individual failure is brief, because the system starts treating the instability as a characteristic of the site, not as a one-off incident, and adjusts the crawl rate downward more persistently. Trust recovery in that scenario isn't immediate once the problem is fixed: Google needs to observe a sustained period of normal responses before raising the crawl rate back to its previous level, so a problem fixed today can still echo in crawling for days or weeks afterward.
Why Search Console isn't a real-time monitoring tool
Search Console's Crawl Stats report shows aggregated data with a delay, typically one to several days, and it's built for reviewing trends, not for catching an outage while it's happening. Relying only on that report to find out about a server outage means, in practice, finding out once enough time has already passed for the problem to have had real impact on users and on crawling, with no chance to react while the incident was still active.
Real uptime monitoring is a separate, complementary category of tool: a service that makes periodic synthetic requests (typically every one to five minutes) from several geographic locations, and alerts immediately (email, SMS, push notification) the moment it detects an error response or a timed-out request. The practical difference is hours of reaction time: with real monitoring you can act while the incident is happening; checking only Search Console afterward, that window has already closed.
Minimum recommended check for a small business site:
- Frequency: every 5 minutes
- From at least 2-3 different geographic locations
(to rule out a one-off failure being just a single network path)
- Immediate alert (not daily or weekly) on error code or timeout
- Check the real HTTP status code, not just whether "something loads"
(a generic error page returned as a 200 goes unnoticed
by a check that only looks for a response)Hosting factors that determine real reliability
A site's stability depends heavily on where it lives, and not every kind of hosting offers the same guarantees. On shared hosting (several sites from different clients on the same physical server, sharing CPU and memory resources among all of them), a traffic spike or a heavy process from a different client on the same server can degrade performance or cause errors on a site that, on its own, hasn't changed anything at all: this is the phenomenon known as the "noisy neighbor" effect, and it's one of the hardest causes of intermittent instability to diagnose from the outside, because the symptoms appear and disappear with no apparent connection to anything the site itself has done.
For sites with meaningful traffic or a real business dependency on availability, having a public status page, where active incidents and their resolution get communicated, doesn't prevent the outage itself but reduces the reputational cost: proactively communicating that a known problem exists and where its fix stands builds more trust than silence, both toward real users and toward the internal handling of the incident.
Frequently asked questions
How much downtime starts to really affect crawling?
There's no exact published threshold, but the most clearly documented case is robots.txt: more than 12 hours with no valid response makes Google assume the whole site is blocked. For other 5xx errors, the effect is progressive: the greater the volume and duration of errors, the bigger the crawl-rate reduction and the longer it takes to recover afterward.
Is checking Search Console weekly enough to catch availability problems?
Not to catch the incident while it's happening: Search Console shows aggregated data with a delay of several days, useful for confirming the impact after the fact but useless for reacting during the outage. That requires dedicated uptime monitoring with immediate alerts.
Does moving to a more expensive host guarantee better availability?
Not automatically: what determines reliability isn't just the price but the level of resource isolation (whether the plan reserves dedicated CPU and memory or shares them with other clients), whether real redundancy exists, and the technical support available during an incident. It's worth reviewing those specific features, not just the plan's marketing label.
Is a single isolated 500 error, seen once, cause for concern?
In the vast majority of cases, no: a one-off, isolated error with no repetition gets absorbed with no lasting impact on crawling. It's repetition and persistence over time, not a single incident, that makes the crawling system start treating it as a characteristic of the site instead of a passing anomaly.