500 Error codes can impact Google Indexing; All you need to know

Google begins removing web sites from its index list in reaction to 500 error response codes. John Mueller discusses what causes Google to begin removing web sites from their index in reaction to 500 faults. Google’s John Mueller responded to a query regarding when 500 incorrect response codes may lead Googlebot to scan sites less often.

Mueller described instances in which 500 response codes have no effect on crawling and when they may cause pages to be removed from Google’s index.

What is a 500 Error Code?

Servers provide various status response codes to browsers and bots. These response codes indicate whether a page request was successful, forwarded to another page, blocked, or failed.

-The 200 response code indicates that the web page request was successful.
-The 500 response code indicates that the web page request was failed.

The 500 response code is formally known as:

"500 Internal Server Error"

Other variants of the 500 response code indicate a specific reason why a web page request was failed.

The 500 response code is very general, indicating that the page request was failed. However, it is not always the case that the server was unavailable.

Accidentally Blocked Googlebot and Caused 500 Errors

The individual who raised the query said that their content delivery system was designed to prevent malicious bots while enabling legitimate bots and human users to access the site. Everything was OK until they changed their “server monitoring suite,” at which point the prior bot blocking filters and settings were apparently not transferred over to the new system. Following that, they observed that Google Search Console began reporting 500 error response codes as a result of Googlebot scanning.

When the individual had completed recounting what had occurred, he or she posed the following question:

“The question came up from our technology team whether this represented real user impact and why we would look specifically at Googlebot and not real user metrics to prove that there’s an issue here.

So, given that context, I have a few questions.

The first is just to get the technology concern out …of the way.

From your perspective, how does Googlebot view 500 series errors and could you give any clarity on established …thresholds at which point Googlebot will …crawl source content …less, based on those errors?”

Google Search Console and 500 Error Responses

It’s a good idea to keep an eye on the Search Console for 500 Error Responses because they’re a good sign that something isn’t right.

It might also imply that the server is under severe pressure, such as from too many bots, and hence unable to deliver web pages. In this example, it appears that the issue was caused by a misconfiguration in the Content Delivery Network (CDN), which prevented Google from accessing web sites accidentally.

How Googlebot Indexing is Affected by 500 Errors

Offering an informative description of the actions Google takes in reaction to 500 error codes, as well as how continuous mistakes may result in those web sites being removed from Google’s search index, Mueller explains.

Google’s response:

“We don’t have any strong thresholds on that.

But essentially what happens with 500 errors is we’ll try to retry them.

And if we continue to see …the 500 errors then we will …slow down crawling.

And if we continue to see that there are 500 errors then we will drop those URLs from the index.

So that’s something if every now and then individual pages have a 500 error, it’s like no big deal.

We will retry them, they’ll remain indexed and the next time we retry them that’s fine.”

Mueller goes on to explain that the reason Googlebot slows down crawling is because Googlebot is causing the server to overload.

“But if a large part of a site consistently has 500 errors and we might assume that maybe we’re causing the problem and we’ll slow down crawling of the whole site and at some point we’ll say well, it looks like these pages are really gone, we’re going to drop them.

So that’s essentially the effects that you would see there and if you’re talking about a large site and wondering like what percentage of 500 errors is okay… I don’t know.

My feeling is if you’re seeing something more than one percent then that sounds like something is kind of broken and probably would be something where we would start to slow down.

But I don’t think we have any hard thresholds where we’d say, this many requests and this many errors means this much slowing down.”

Monitor Search Console for 500 Errors

It is up to each publisher to determine their daily schedule. For some, checking the search console once a week is sufficient. It may be reassuring to some to check Search Console once a day in order to respond to unanticipated events.

In any event, 500 error codes are something to keep an eye on in Search Console. Large-scale 500 mistakes, as John Mueller said, might be a red warning that something is amiss and requires quick attention.