Can Google Ever Spider 29.5 Million Pages on One Site?

Sep 13, 2007 - 9:18 am 3 by

A WebmasterWorld member asks how long Google will take to spider all 29.5 million pages on his website. He just launched the site 2 weeks ago and is hoping that *all* pages get spidered.

That's a generous request of Google. After all, too much content being released at once can be risky, as one forum member says:

Matt Cutts discusses the possibility of triggering a penalty if too many pages are released at once.

That being said, I'd go for it because if you've created a quality site you're going to attract the links you need to start climbing the SERPs.

However, so far, 2,000 pages have been crawled. If the Googlebot goes on a path of spidering all these pages, it will take awhile.

Based on the current indexing routines of your site (143 pages per day), I'd say you have about 56.52 years in the queue. ;)

Maybe it will. After all, large sites like only has 18.7 million pages indexed. It will be hard to compete with authoritative sites even if they're not in the same space.

But the site, which appears to be a city directory of some sort, will have its challenges. It's rare to get that many pages crawled. And moderator pageoneresults adds that a site like Amazon is " is seeing upwards of 5+ million Googlebot visits daily." On the contrary, the webmaster says that Googlebot has only visited him 22 times since launch.

It's possible for Google to crawl more pages, but I'd also find it hard to expect all 29.5 million pages in the index. As tedster says, it will really depend on the backlinks to the sites. The more quality backlinks and the more quality deep links you have, the better you'll fare.

Forum discussion continues at WebmasterWorld.

This post was composed on September 11th and scheduled for publication on September 13th.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google Core Update Flux, AdSense Ad Intent, California Link Tax & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: April 19, 2024

Apr 19, 2024 - 4:00 pm
Search Video Recaps

Search News Buzz Video Recap: Google Core Update Flux, AdSense Ad Intent, California Link Tax & More

Apr 19, 2024 - 8:01 am
Google Ads

Google Tests More Google Ad Card Formats

Apr 19, 2024 - 7:51 am
Google Search Engine Optimization

Google: It's Unlikely Your Rankings Dropped Because You Have Two Websites

Apr 19, 2024 - 7:41 am
Google Search Engine Optimization

Google: Indexing API May Work For Unsupported Content But...

Apr 19, 2024 - 7:31 am
Google Search Engine Optimization

Google: Are Hyphenated Domains Bad For Google Rankings?

Apr 19, 2024 - 7:21 am
Previous Story: September 26: MSN Live Search 2.0 Arrives