(33)
Last week Tamar wrote about How to Stop Googlebot from Crawling Your Site Rapidly, so I thought I write about the opposite. How can you induce GoogleBot into crawling your site.
Although there is no magic shot that guarantees inducement of the crawl you can always increase your chances of that crawl in a few ways.
- Submit a Sitemap file to Google Webmaster Tools
- Make sure you have plenty of links pointing to your site from pages that are already in the Google index
- Build out a real sitemap for users
- Make sure you have a search engine friendly site
- Increase the crawl rate at Google Webmaster Tools to "Faster"
I am sure there are many other ways to help induce a crawl, but here are just a few good ways to do it.
Forum discussion at DigitalPoint Forums.
Comments:
Peter
10/11/2007 12:32 pm
re Increase the crawl rate at Google Webmaster Tools to "Faster" As far as i can see this option is hardly ever available.
Rich09
10/11/2007 01:09 pm
Any information regarding Googlebot crawl rate speed = faster would be of great help. I am new to web design, having started a New Media company I aim to add value by generating SEO friendly website design, researching the concept of a symantec web and contirbuting, where possible, to the development of structured on-line information
Ben Pfeiffer
10/11/2007 03:33 pm
The option to set crawl rate to "faster" is available more often if you have old domain or sites which haven't been updated or doesn't contain any substantial content. You then launch a new site on this domain and sometimes you get a "faster" option but not always. It does work well if this option is available, it will usually hit your site for 3-4 months and then taper to a normal level.
JohnMu
10/11/2007 08:42 pm
As mentioned in http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=48620 the "faster" setting for crawl rate is only available when it is determined that the Googlebot would be ready to crawl the site faster anyway. For sites that are not being crawled yet, this setting will likely not be available. Also, as mentioned, the setting will return to normal after 90 days. Keep in mind that "normal" could be faster or slower than before (it depends on many factors).
Raptorak
02/26/2009 05:59 am
I turned my site to faster crawling but it is not indexing newer pages. For example, my last page added was on Feb. 11th..And it was added almost instantly. Since then it has not indexed any new pages..
Rob Abdul
02/26/2009 02:05 pm
Raptorak, just a thought, Do you have a sitemap.xml? If so have you tried to set the priority of your new page to 1. Does your last page added have a link form your homepage or the highest ranked page on your site? Google in February has been very slow in indexing content. Yahoo Slurp is much faster at the moment.
redown
10/15/2009 03:23 pm
it is quite useful information.i have set my site's crawl rate to faster. And i think its much better than normal because googlebot is indexing my pages faster than before. Its nice and thanks to webmasterteam.
No Name
12/16/2009 05:26 am
Great information. But in google webmaster I still dont have the special setting for the crawl rate?
David Francis
05/30/2010 09:08 pm
I have set my crawl rate to much faster for http://www.ocreef.com and have not seen a significant index, and it has been a couple of weeks now. Does anyone know how soon they would see any differences? The rate returns to normal after 3 months.
HP Bryce
06/05/2010 12:52 pm
It has been over a month since Google has crawled my site http://whitestagcircle.com and I am set to fast on webmaster tools. They just tell me to wait. makes me want to scream. Over one month and counting is too long for a new site IMHO.
Bradley
06/23/2010 11:53 am
My site http://www.123businessregistration.co.za for peeps who want to register a cc, also gets slow crawl rates. I found that making sure all of your links are not broken cause this can affect how Google crawls your site.
Brads
07/21/2010 04:01 pm
Another quick question. How do you know when last Google-bot crawled your site?
wally
08/05/2010 11:56 am
I've been getting Googlebot visits everday to my blog because I add content everyday but I just finally added a sitemap to my blog for the first time. How do I know if the sitemap is working correctly? Thanks for this post.
Pradeep
09/03/2010 11:50 am
@Brads (Posted by Brads at July 21, 2010 12:01) Webmaster tool can show that info under crawler access. But, you can do a little tweak to it if you want to instantly know when googlebot or any other major search engine bot crawled you. How to do that? Check this, http://www.expertsguide.info/2010/08/send-mail-when-googlebot-crawls-a-webpage/
Jake W.
10/30/2010 05:52 pm
My website is not getting crawled as often as I wish but I'm working on it. Thanks!
David Pye
11/28/2010 06:26 pm
Thanks for the advise but my site http://www.retailzone.co.uk does seem to be getting crawled still
Ceri_davies
03/08/2011 09:35 am
i am also having a few issues with my website www.mosmanhouseminders.com.au
Malay Kothari
03/12/2011 01:16 pm
i am trying to do the same for my site www.rushabhacademy.com
JKuro
05/26/2011 08:12 am
does this googlebot service cost something?
Tits
05/28/2011 01:48 am
Wow, the whole articel was based on how to increase crawl rate, but all you did is list the one's everyone knows then say "I am sure there are many other ways to help induce a crawl, but here are just a few good ways to do it" Which basically means you don't know what they are and you're providing nothing more than ABC content.
Pakidil.com
06/23/2011 07:17 am
update your site with good contents and google surely crawl your website with in a day . Lots of Pakistani Focusing on this Thing and myself haven't got any crawl problem yet but make sure your contents is unique not copy paste.
cvaliente
06/29/2011 06:37 pm
Thank you for the tip, I am currently promoting 3 products launches through 3 of my blogspot webpages. I recently made an update on the content but Google has not crawled it yet and I would like it to get crawled soon. I enhanced the on page optimization on it so it very important. Below are my 3 webpages: Feedback would be helpful...Thanks List Eruption Bonus Review Commission Vigilante Bonus Google Siphon Bonus
waqas umar
07/22/2011 12:23 pm
thankyou , its helping, hope to get some way from here:)
Dial Kbb
10/29/2011 03:23 pm
some one help me to list my website on google first page in local area, www.hostingaffair.com kindly mail me with Google Key word for listing, if possible. aslo looking to sponser of the same, if someon elooking. regards DialKaro email at: dialkaro@dialkaro.in Gtalk Id: dial.kbb@gmail.com, Yahoo Id: dialkaro_sms@yahoo.com www.dialkaro.in www.hostingaffair.com http://sms.dialkaro.in
Infonep
11/04/2011 05:25 pm
Thank you I hope I now can have bot crawl over my site.
Pandhi
11/28/2011 12:41 pm
i want to make google crawling for my website http://www.pandhimedia.com
Metalwork
12/20/2011 04:16 am
I messed up! Changed my title page by mistake. I am now patiently waiting for google to crawl my site: www.SEOIC.com
Alex Harrild
01/05/2012 10:10 pm
me too!! just launching and now waiting for my next google crawl!! please soon!! www.comamosalgo.com
Klsdf Ol
01/25/2012 07:41 am
I have set my crawl rate to much faster for www.ultragames.dk and have not seen a significant index :(
Dilhan Maduranga
02/10/2012 04:51 am
I too wanted to index my site www.mymarket.lk . I used web master tools, site was indexed "Fetch as google-bot" is really helpful. Then I found out using seotoolscentral that the site to be more optimized. I changed as the tips on the site. ' That was some days ago, but still http://www.mymarket.lk/web/home/index.php is not reindexed. Isn't there a meta tag or anything to say the crawler to come and check my site frequently?
Guest
03/01/2012 06:40 am
Google did not crawl my site in 6 month... Please help http://fedeopskrifter.dk
Princes S
03/14/2012 08:25 pm
Google did't crawl my site.. http://www.visa-abroad.com
shafinbest
12/02/2012 01:53 am
Crawling is very important. We know. But how to get our site crawled is a question. This post covers some good points describing the method. :)