500/503 HTTP Status Code On Robots.txt Can Remove Your Site From Google Search

Dec 14, 2022 - 7:21 am 1 by

503 Google

Gary Illyes from Google said on LinkedIn that if your server returns a 500/503 HTTP status code for an extended period of time for your robots.txt file, then Google may remove your site completely from Google Search.

This is even if the rest of your site is accessible and not returning a 500 or 503 status code.

It is not just a 500/503 HTTP status code that you need to worry about, it is also an issue if your site does those network timeout issues.

Again, it has to be for an "extended period of time," which was not defined, but I assume it is more than just a day or two.

Gary wrote, "A robots.txt file that returns a 500/503 HTTP status code for an extended period of time will remove your site from search results, even if the rest of the site is accessible to Googlebot." "Same goes for network timeouts," Gary added.

Gary referred to the HTTP docs and added, "if we can't determine what's in the robotstxt file and the server doesn't tell us a robotstxt file doesn't exist, it would be way more hurtful to crawl as if everything was allowed (eg. we might index martin's awkward hat pictures accidentally)."

We know that Google recommends using a 503 server status code for when your website goes offline or down temporarily for less than a day (hours not multiple days). If it goes offline for longer, than try to upload a static version of your site in its place.

Just be careful with long down time - not that you likely have a choice.

Forum discussion at LinkedIn.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google AI Overviews, Ranking Volatility, Web Filter, Google Ads AI Summaries & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: May 23, 2024

May 23, 2024 - 10:00 am
Google

Google Robots.txt Parser Updated In GitHub

May 23, 2024 - 7:51 am
Google Ads

Google Ads Referral / Affiliate Program

May 23, 2024 - 7:41 am
Bing Search

Bing Search Is Down; Takes Down DuckDuckGo, Ecosia, ChatGPT Search & More

May 23, 2024 - 7:35 am
Google

Google Tests Replacing Related Searches With People Also Search For?

May 23, 2024 - 7:31 am
Google

Google Search: Sources Across The Web Without Images

May 23, 2024 - 7:21 am
Previous Story: Google Again: Time Spent On Page Is Not A Ranking Factor