Fetch As Googlebot Shows Only 100Kb But Real Googlebot Gets More

Oct 14, 2009 - 8:33 am 1 by

Google added the fetch as Googlebot feature the other day and now people are really beginning to explore it. One topic I have seen come up was why is the Fetch as Googlebot feature only showing up to 100Kb of the page it is fetching? Does that mean Googlebot only crawls up to a 100Kb of a specific page?

The quick answer is no, Googlebot does index more than a 100Kb, but the fetch feature only shows up to a 100Kb.

Historically, Googlebot at one point only indexed up to 100Kb. In fact, some time in 2006, the Google cache showed over 100kb of the page, which put the 100Kb maximum page size limitation to rest.

Google does indeed index pages larger than a 100Kb, especially in the days of higher bandwidth. But in terms of the Fetch as Googlebot feature, for speed purposes, it only grabs 100 Kb for this tool.

JohnMu of Google said in a Google Webmaster Help thread:

As far as I know, this is a limitation of the Fetch as Googlebot feature, so I believe more or less the only difference between a real Googlebot and this feature. The main problem is that arbitrary file sizes would bog down the Webmaster Tools user interface so we had to draw a line somewhere.

Forum discussion at Google Webmaster Help.


Popular Categories

The Pulse of the search community


Search Video Recaps

Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: May 24, 2024

May 24, 2024 - 10:00 am
Search Video Recaps

Search News Buzz Video Recap: Google Ranking Volatility, Ads In Google AI Overviews, Sundar Pichai Interview, Heartfelt Helpful Content & More Ad News

May 24, 2024 - 8:01 am
Google Search Engine Optimization

Google: The Site Reputation Abuse Policy Enforcement Not Yet Algorithmic

May 24, 2024 - 7:51 am
Google Search Engine Optimization

Google Search Can Now Index Electronic Publication (EPUB)

May 24, 2024 - 7:41 am

Directory Of Embarrassing Google AI Overviews

May 24, 2024 - 7:31 am
Web Analytics

Google Analytics Real-Time Reports Adds Users In Last 5 Minutes

May 24, 2024 - 7:21 am
Previous Story: Microsoft's Bing Hiding Search Results For Perez Hilton