Webmasters and SEOs have new reading material for this weekend. Google has published a very comprehensive and detailed Robots.txt Specifications, Robots meta tag and X-Robots-Tag HTTP header specifications and how to control crawling and indexing by GoogleBot.
Tedster said he learned at least one new thing from this new resource. He said:
Google will look for and obey an FTP robots.txt file located at ftp://example.com/robots.txt
PageOneResults added a highlight on:
Redirects will generally be followed until a valid result can be found (or a loop is recognized). We will follow a limited number of redirect hops (RFC 1945 for HTTP/1.0 allows up to 5 hops) and then stop and treat it as a 404.
What did you learn?