Ranking Software & Google's Opinion

Aug 15, 2008 • 8:43 am | comments (20) by twitter Google+ | Filed Under Search Engine Optimization Tools

A couple weeks ago, Google slightly changed how they show the Google search results, which resulted in several search ranking software packages to not function properly anymore. I reported this under the title of WebPosition Ranking Software Being Blocked by Google? early on and Scott Goodyear from WebPosition commented here and also at WebmasterWorld with his two cents.

Scott took offense to me using the word "blocking" in the title. The truth is, I posed it as a question and Google was not technically blocking them, just yet. But will it happen? It seems like it.

Matt Cutts of Google has commented (once again) at WebmasterWorld on his stance on ranking checking tools, saying:

I wouldn't be surprised if our bot detection continued to get better over time for many types of software that scrape Google. We'd prefer to use that server capacity for real users, not someone who wants to check how they rank for tons of queries going down hundreds of pages.

Does this mean that Google is now going to try really hard to prevent scraper based rank checking software? Seems like it.

Google's JohnMu also commented at our site, explaining some of the history:

I think the information given in the Webmaster Guidelines and the TOS is fairly straight-forward:

"Don't use unauthorized computer programs to submit pages, check rankings, etc. Such programs consume computing resources and violate our Terms of Service. Google does not recommend the use of products such as WebPosition Gold™ that send automatic or programmatic queries to Google."

and "5.3 You agree not to access (or attempt to access) any of the Services by any means other than through the interface that is provided by Google, unless you have been specifically allowed to do so in a separate agreement with Google."

Matt Cutts also commented in a thread Scott started in the Webmaster Help groups.

I am not going to get into the argument, if this should be allowed or not. I would hate it if people wrote scrapers to constantly scrap my pages, oh wait, they do. It is a waste of my bandwidth and can impact the speed of our page loads. At the same time, there are easier ways to get this data that might be more in accordance with the TOS.

There is also a big debate on why SEOs need to bother with ranking reports.

I'll leave the discussion in the air.

Forum discussion at Google Groups, Sphinn and WebmasterWorld.

Previous story: Google Opens AdSense For Feeds


Kevin Doory

08/15/2008 02:17 pm

""At the same time, there are easier ways to get this data that might be more in accordance with the TOS." Do you have examples of this?

Barry Schwartz

08/15/2008 02:23 pm

http://code.google.com/apis/ajaxsearch/ But if you have the old API stuff, you are set. :)


08/15/2008 05:16 pm

I don't use my old Google API key any more. I wonder if I could sell it? :)

Matt Cutts

08/16/2008 05:44 am

Kevin Doory, don't forget that the referrer data from your server logs also essentially tells you all the search queries for which you receive any real users from a search engine.

Gert Mellak

08/16/2008 09:08 am

I completely agree that the users should be the ones consuming our servers' bandwidth. Why doesn't Google Webmaster Tools offer a feature to enter a list of keywords and follow their rankings over time in google.com, google.at, etc.?


08/16/2008 05:41 pm

If you have a site of blog, and you want to use Google's Adsense, You will include Google's analytics and Google's Webmaster tools... Do you think you play around with Google this way?

Chris Peters

08/16/2008 07:03 pm

Arguments on whether we should be doing rank reports or not, I agree with Gert's question. There is clearly demand for automated rank checking. So why doesn't Google give us some way to do it that's in line with their TOS?

Doug Heil

08/17/2008 08:48 pm

"There is clearly demand for automated rank checking." Really? I don't know whether I would call it a "demand" at all. I think it's more of some SEO's not knowing how to teach their clients on the value of such reports. Considering the value is zero, it makes things easy... I thought. I also think the SEO's who charge some kind of monthly fee feel like a rank report is part of that, even though the client could actually do that rank report themselves if they felt such an urgent need. That statement above was made, so one should also note that many other SEO's have not used a rank check software program in years and years, and for all the many reasons already stated.

George Bounacos

08/18/2008 01:20 pm

This is a great post because you shut down the issue of right or wrong. It boils down to "How is this addressed in the TOS?" With so many vague areas every engine doesn't define, when one does say, "Don't do this", that pretty much shuts down the argument. As to why Google doesn't give these SEOs a similar function, the answer may be as simple as "it's a business decision, and they don't want to." The only time I've had issues with this is with clients who don't want to be educated. They want to know that they rank top 10 for blue widgets. You can tell them how SERPs are built and influenced, and it doesn't matter. They want to type the phrase *blue widgets* without any operators and see their site at #1. I tend not to keep those people. ;-)


08/18/2008 02:03 pm

Google is saying the when webmasters use scrapers to access Google they waste Google's resources. I agree! Does Google agree that when they sent out GoogeBot to crawl a webmasters entire website they are also wasting the webmaster's resources if a sitemap exist? Now that we have sitemaps, all they really need to do is index the URLs listed in the sitemap found in robot.txt file based on the priority given. If all can not be indexed drop low priority url's first.


08/18/2008 02:44 pm

Doug, there is clearly a demand to check rankings. My team has been able to discover many technical issues with our website that would not have been apparent if we weren't watching the trend of rankings for specific areas of our site. We also have internal stakeholders that demand to know where we are ranking. Its not an issue of right or wrong, but how the engines prefer to manage this. How about an IP address dedicated to rank checking software for a fee? We would gladly comply with a realistic solution.


08/18/2008 05:01 pm

jim, I totally agree...a rookie mistake is to not check ranks to set a baseline that way you can do active SEO and see what is working and scale appropriately...


08/18/2008 09:09 pm

I see these position reports as valuable. Lets say that there are 20 phrases that generate the bulk of sales for a website. The drop in rankings for any one of these phrases could seriously impact a bottom line. Why wouldn't you want to know how these phrases rank in Google? Sure you can manually do 20 searches and take a look, but when you have numerous clients all needing to see this data, doing it by hand is near impossible as there is only so much time in the day. I agree that analytics, etc, are all important, but with a ranking report, it allows you to see trends in listings, etc.


08/21/2008 07:26 am

If I use a tool to check 10 pages for my ranking, will I not also use the same amount of Google resources doing it manually? Perhaps the software companies should put a ceiling on the ranking checking tools out of respect for Google and others. That being said, if it goes against Google policy anyway, I think from now on ill do in manually! After all, Google has been REALLY good for my business.

Doug Heil

08/23/2008 12:07 am

Hi Jim, You wrote this: "Doug, there is clearly a demand to check rankings. My team has been able to discover many technical issues with our website that would not have been apparent if we weren't watching the trend of rankings for specific areas of our site." Now see; that's what I mean Jim. I'm not getting that line of thinking at all. I wish I could get it as that is the same line as many in this industry, but I just can't. Impossible. First off; Google has stated for years and years to "not" scrape their serps. Period. They have been extremely clear about that so any program or firm who insists on it is violating their TOS and subject to penalties. period. Second; What you just stated above is more easily seen in your stats. You don't have to violate a site's TOS either. I can find trends and abnormalities for keyword phrases from the stats and logs of the site. Period. No one has been able to answer my questions of why you have to run auto rank check reports in a satisfactory way yet. Sorry.


08/26/2008 01:29 pm

This is just big brother Google bullshit. SEO are trying to make a living like everyone else . Matt Cutts can go get fucked


08/28/2008 10:31 am

Hi Doug, I am certainly more pro automated rank checking than anti, but I am interested in a couple of things you said: "I can find trends and abnormalities for keyword phrases from the stats and logs of the site. Period." and "No one has been able to answer my questions of why you have to run auto rank check reports in a satisfactory way yet. Sorry." How do you find how well your sites are ranking compared to competing sites from site stats and logs in a regular and time efficient manner? For instance, say one of your sites also sells the proverbial blue widget - how do you know where you are ranking in comparison to the competition and remain on the ball should a ranking drop occur? I'm of the opinion that it is a fair assumption to make that if you rank lower than others for a key phrase, they will generally be attracting higher levels of traffic. Yes, once you have attained a #1 ranking and you see traffic coming through as a result of it, you could run the query yourself to see your position, but surely not day in, day out? How do you cope monitoring multiple key phrases, on multiple sites for multiple clients without spending (read: 'wasting') copious amounts of time doing it? Surely an automatically generated snap-shot that you can glance at upon reaching your desk at 9AM in the morning is easier than manual querying? At least by 9:05AM you'll have some idea of whether you're still in the game (and presumably attracting more traffic than those below you) or have fallen and can therefore concentrate on finding out why. Auto rank checking software often provides historical data too so you can review rankings over time for whatever reason. Do you log such information manually in a book or spreadsheet? Viewing trends and abnormalities for keyword phrases over time is one thing, but actually taking into consideration competitor rankings and your own competitiveness is completely different in my opinion.

Joshua Steimle

09/30/2008 09:31 pm

As everyone knows, there are a lot of companies that violate Google's TOS on a regular basis. In my case, I'd love to violate it, but I can't bring myself to do it because I think it's dishonest to enter into an agreement and then break it. So the end result of this policy is that those people who would like to do the right thing are punished, and those who don't care get rewarded. If Google truly wants to "do no evil" they could start by providing means for those who want to do the right thing to do it so that not only those willing to ditch their integrity get ahead.


12/08/2008 02:47 am

When I search "webposition gold review" in google, I found many seo experts remove webposition from their list recently. like: http://www.bradsoftware.com


06/15/2011 04:30 pm

I would love to see an update discussion on this. What are SEOs' opinions of and Google's real position on section 5.3 of their Terms of Service.   Is Google REALLY trying to stop us from using Rank Checkers - or is there some compromise possible ? Thanx !!

blog comments powered by Disqus