Don’t Use 403s or 404s for Crawl Rate Limiting, Says Google

Google has noticed an increasing number of website owners and content delivery networks (CDNs) using 4xx client errors to try and limit Googlebot’s crawl rate. In response, Google has advised against this practice and instead recommends using its documentation on how to reduce Googlebot’s crawl rate.

Client errors, such as 4xx errors, are signals from the server that the client’s request was incorrect in some way. They do not suggest any issue with the server itself. While 429, which stands for “too many requests”, is the exception, sending other 4xx errors as a means of rate limiting Googlebot can have significant consequences.

All 4xx HTTP status codes, except 429, can result in Google removing content from search results. Additionally, if a website owner serves their robots.txt file with a 4xx HTTP status code, Google will treat it as if it doesn’t exist. This can cause problems if a rule disallowing crawling is in place, as it will no longer be enforced.

Instead of using 4xx errors for crawl rate limiting, Google recommends using either Search Console to temporarily reduce crawl rate or returning a 500, 503, or 429 HTTP status code to Googlebot when it’s crawling too quickly. The search giant has extensive documentation available on how to reduce Googlebot’s crawl rate and how it handles different HTTP status codes.

Google advises website owners and CDN providers to follow these guidelines and to avoid using 4xx errors for crawl rate limiting, as it can have unintended and significant consequences for their website’s visibility in search results. For additional tips or clarifications, website owners can reach out to Google on Twitter or post in the company’s help forums.