500/503 HTTP Status Code On Robots.txt Can Remove Your Site From Google Search

Dec 14, 2022 - 7:21 am 1 by

503 Google

Gary Illyes from Google said on LinkedIn that if your server returns a 500/503 HTTP status code for an extended period of time for your robots.txt file, then Google may remove your site completely from Google Search.

This is even if the rest of your site is accessible and not returning a 500 or 503 status code.

It is not just a 500/503 HTTP status code that you need to worry about, it is also an issue if your site does those network timeout issues.

Again, it has to be for an "extended period of time," which was not defined, but I assume it is more than just a day or two.

Gary wrote, "A robots.txt file that returns a 500/503 HTTP status code for an extended period of time will remove your site from search results, even if the rest of the site is accessible to Googlebot." "Same goes for network timeouts," Gary added.

Gary referred to the HTTP docs and added, "if we can't determine what's in the robotstxt file and the server doesn't tell us a robotstxt file doesn't exist, it would be way more hurtful to crawl as if everything was allowed (eg. we might index martin's awkward hat pictures accidentally)."

We know that Google recommends using a 503 server status code for when your website goes offline or down temporarily for less than a day (hours not multiple days). If it goes offline for longer, than try to upload a static version of your site in its place.

Just be careful with long down time - not that you likely have a choice.

Forum discussion at LinkedIn.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
- YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google

Google Search Bitcoin Charts Gone - It's A Bug

Feb 9, 2025 - 11:45 am
Search Forum Recap

Daily Search Forum Recap: February 7, 2025

Feb 7, 2025 - 10:00 am
Search Video Recaps

Search News Buzz Video Recap: Google Volatility, Google Ads Quality Update, Google AI Overviews Updates & Other Search Engines

Feb 7, 2025 - 8:01 am
Google Search Engine Optimization

Google Says People Know When Author Bios Are For SEO Purposes

Feb 7, 2025 - 7:51 am
Google Ads

Google Ads Keyword Planner New Auto-Organize Keywords To Ad Groups

Feb 7, 2025 - 7:41 am
Google Maps

Google Business Profiles Reinstatements Now Include Case & 5 Upload Documents

Feb 7, 2025 - 7:31 am
Previous Story: Google Again: Time Spent On Page Is Not A Ranking Factor