Hi Barry,

I was debating if I should write about this or not. I had found the excerpt you quoted above plus another on sitepoint:

I have the paid version of webposition and the pro version on WebCEO and since Friday neither have been working for UK position checks. Seems like Google have changed something to stop people using automated software. Hopefully both pieces of software will find a fix but until then

I was experiencing the same problems with WebCEO in NYC and Toronto but I did find a fix which was pretty simple. I would think most people would figure out how to circumvent this.

So this is something new to you all? Do you honestly think that a quality search engine wants robots rushing through their servers? Hell no people. My goodness; why do you have to do rank reports anyway? Ask me how many reports like that I have done in the last 7 years? NONE. Focus on stuff that actually benefits your clients. If a client totally needs to know a rank, it's damn easy to go to Google and do a search, right? It's amazing to me a SEO actually includes a rank report in their services.

Wow - we agree on something Doug. If only I could get my clients to think this.

client: "I don't care if my traffic and sales have doubled - we are down 2 places on our main keyword, what is going on Dave?"

Further; when that client does do a search, that doesn't mean I would see the same damn position on my end, right? Don't ya all know there are many variables involved with doing a search these days? I want to know exactly what the infatuation is with a serp position on a keyword phrase. Please explain why you all put money into a automated rank check program these days when stuff like:

local search
geography location
universal search
whether or not you are logged into google services

on and on and on with many variables.

Why would you think these reports are doing your client or even you any good whatsoever? Give it up people.

Frankly; it's astonishing someone like Barbara Cole bothers with WP or any other program. I am totally amazed by this. I didn't know the SEO industry still thinks a rank check tool is something a client needs. I'd rather listen to fingernails against a chalk board these days than listen to this industry and all the rubbish.

Hi Dave; I understand that totally. It's been real easy for me to explain as my post above details. What the heck good does it do your client since he/she is seeing something different than you are? Why can't he simply do the damn search himself? What is the actual benefit of you using some silly rank check software for this?

Do you see? I will never understand this. Never.

Just to clarify. I personally do not run ranking reports. All I am doing here is reporting on a thread that people might find interesting.

lol Barry; I didn't think you did as I had thought you stated that awhile ago.

I'm quite shocked so many still do.

Ranking reports are something that I've provided my clients with for years. However, I do emphasize that they shouldn't put much if any weight on the rankings that these reports detail. I always stress that what matters most is traffic and conversions. Most if not all of my clients understand this. So Dave I agree with you that reporting on rankings are useless for the most part. It's just a small value added to the overall monthly package that I provide my clients with.

Hi Brendan, But what "small value" would that be considering the details I posted above? I'm really trying to understand the mindset here is all. :) Why would anyone think they need this small value added to any package at all? Why not educate the client as to why they are just not needed and don't hold a single piece of value?

You or the client can do a manual search on google for:

pretty red widgets

You can both see different results on that phrase. If that client isn't on at least page three,.. no need to continue searching further as you already know you need more work. If a phrase is on page 5 or page 10, what's the difference? LOL And the fact that you may see the phrase on page 5, but your client sees the same phrase on page 7,... exactly what have you gained as far as value?

Please understand I am trying hard to see the other side of this stuff. I'm not a slow learner either, but I'm having a hard time with this. :)

All of my clients are educated about the triviality of ranking reports but it's still part of the monthly package. It's kind of like the keywords meta tag you still have on ihelpyou.com. They're useless but they're there.

If Google blocks all the rank position checkers they'll lose substantial ground in the bogus search market share reports published by Compete, comScore, Hitwise, and Nielsen (who only look at number of searches performed).

Yes ago I got banned by Google for running these reports. I then phased out ranking reports years ago for clients and haven't looked back. We instead provide web analytics & conversion training. I still get requests for them and if a prospective client insists on them included in our services I will not take on that client if it's that important to them.

I am not surprised to see Google even ban the IP address for say 24 hours of people that are using any type of rank checking software. The software has to leave some type of foot print for google to detect it and of course they can see the IP address that is making all of the requests. I do not see the point in showing a client some report that can easily be manipulated. Just do a search and they can see for themselves.

Very nice Ben and Steve. Now that is what is important and actually has value.

Brendan; it's like the keywords meta tag? You must be joking. It's nothing like that at all. I'm not selling a package to a client based on that meta tag, like you are selling services based on a rank check report... even if a so-called "small value". You really should start educating your clients Brendan.

Michael; so true. Considering that many keyword research tools are messed up as they combine any of every kind of search including software/robots searches, etc.

It's surprising that even though Google specifically states in their guidelines to NOT use automated tools on their servers, ... many still do. Let's see; how many years has that WP mention been in the G guidelines?... quite a few I think.... since about 2003 maybe?

I even recall the fight between WP and Google waaaaay back when. Anyone remember TopDog? lol Google did not blink an eye at that time either. If you had a site which was a very busy site in some way, would you allow automated tools to run a muck through your site daily?.... millions of bots?.... I don't think so. Why would G? Why would anyone?

AWR still works fine.

I knew that I would get the 'I can't believe they still run ranking reports' comment. You are right, I am not so behind the times and am very, very clear with clients that it is NOT the measurement they should be using for tracking.

That said: in order to track organic, video, or image download or whatever that is not a paid search or online click you need the cooperation of a whole different group in the company. Sophisticated clients have most web marketing run out of marketing organizations with marketing budgets - they don't own IT or the traffic analysis packages. Some are using completely different systems to track marketing campaigns, of which search is a part, like marketfirst or eloqua. The in-house SEO people at these clients are in the marketing divisions now and we work with them. It is the sign of a mature company if SEO and Paid Search are all in marketing - but it creates other problems.

The best way to track organic is to tag the visitor in some way but because it is not tracked say with a dart tag or a Google adwords tag, which can be set up by marketing people at the campaign level, this tracking requires 'help' from a different division.

Executives need something that is highly visible, isn't just about numbers and shows improvement or problems readily - thus rankings.

So, as we mature and move SEO away from the IT/developer crowd into the world of marketing campaigns we lose some of our ability to control the technical side of the business. I would rather it be part of marketing anyway.

Yes; I understand that totally. I highly doubt that most SEO firms out there and some in this thread have that kind of situation though. I'll bet that most are selling package SEO services with rank checking as part of that package. Even with the scene you mention Barb, two people can be looking at the same serp with very different results. I really don't see how that helps them either. Couldn't they simply do a manual search on google to see the same thing? I guess I'm still stumped why people want to go against the guidelines anyway, even when they state to not use auto programs to check positions. Don't these in house SEO's know this as well?

Is this the same with WebCEO tool. Please let me know since I have started using this tool?

Of course. That webceo thang is automated, right?

WPG take me back.. all those reports hehehe..

So glad I don't do that cr*p no more..!
Ben, Doug and with you guys..

LOL @ Gabs. I felt like I had no life when I ran ranking reports. Noooo more reppporrrtts... Glad I came too and realized the there was life outside reports.

Hello everyone. I thought I'd weigh in on this. I've been in the business for 11 years+, run a reputable SEO shop, along with paid search and web analytics/conversion analysis. There are four measures of success, important to clients, which include but are not limited to: organic and paid visibility; resulting traffic (need to prove qualified); resulting sales or leads; resulting profitability. Measuring organic position, and trending that over time can help to determine penalties for IT problems, that Google Webmaster Tools can not tell you. Asking a CEO to "check his or her ranking" in a measured success trend report probably wouldn't go over all that well. Most companies are unable to prove a lead accurately came from organic, even though tracking is in place. That said, the only other form of success that you can prove is "improved visibility".

We produce ranking reports twice a month, monthly traffic, sales and leads data once a month, and recommendations on how to improve things.

In fact, we recently got a client from an SEO consultant that did not track rankings. There was a hack attack on the server, and the SEO consultant did not catch this two months into the decline in rankings. Perhaps if he/she had monitored rankings, they would have caught the problem sooner - and would have kept the account.

Because we monitor, we get more accounts this way. We also are careful to monitor more than visibility.

At the end of the day, you should know your core, profit-earning terms. If you can not improve visibility for thos core terms, you should not be doing SEO, or paid search.

A footnote to earlier post: I've been in touch with WPG, which I've used "responsibly" for eight or so years. They indicate the HTML output is different in Google, and thus, they're working on the update. As Barb noted above, there is no ETA on the update.

As an alternative, you can use AJAX API which will provide up to 20 search results. Second, you can also download Trellian's SEOToolkit, which includes a RankChecker component. Our office was able to get Google organic rankings yesterday.

Hi Laura Thieme; I just have to disagree totally. Why is that many of us have not used some silly rank check software in at least 5 years?.. me included? I know many. Don't say we don't deal with bigger clients either... as I do. You state this:

"Measuring organic position, and trending that over time can help to determine penalties for IT problems, that Google Webmaster Tools can not tell you. Asking a CEO to "check his or her ranking" in a measured success trend report probably wouldn't go over all that well."

Penalties? Well, if in fact you are reviewing your stats program or your log files on a daily basis, etc, wouldn't you or that CEO notice when referrals from google or se's went down? If so, doesn't that tell you that there just may be a problem? Sure it does.

It makes no difference what justification or how much rationalizing you all do with this, but you cannot make a good enough argument for me using a rank check software robotic and automatic program that runs around to the se's using up all their resources. And for what? Nothing.

Laura; That CEO and yourself can spot any darn prob in a heartbeat, and even do so quicker simply reviewing your daily stats or logs. My goodness. :)

Why do you all insist on going against what has been stated in the google guidelines for years now? NO WEBposition should be used. That's what it says in black and white.

I've got to go further.

This SEO industry.. or whatever it is... needs to get away from the crap non-sense of making the general public think that we are all about a rank. That's crazy thinking people. I'm just wondering out loud here, but how many of you will walk away from a potential client?

What if you review a site that you may want to help and you actually think there is NO WAY the site will sell anything? Would you walk away if that owner does not want to rebuild the site? If not, why not? Are we all about the rank and not much else matters? Are we? I'm not.

I've walked from many a site over the years. If not because the owner was soooo fixated on a rank, it was because the owner simply thought that all would be just dandy if only he were on the first page of google for.... X phrase. The fact is that owner's site wouldn't have a prayer of selling anything even if he were on that first page for that X phrase.

How many would take the money anyway? How many?

It appears that many in this industry still are not educating in what the actual value of SEO is and is not. You couldn't be doing so, otherwise you would see clearly that rank checking has NO value whatsoever. You can get the same things by viewing your stats or logs...and more. You can get the same things by using the g webmaster tools. You can get the same things by knowing exactly when and how your positions must have taken a tumble on some phrases because all the sudden your referrals on those phrases are now nothing. Why the hell do you need some silly rank check software to tell you that?

Why also do you need that same software when your client in North Dakota is seeing his beloved phrase on page 5 and you see it on page 2..... why the heck, Huh?

Sorry. I can get off sometimes. :D

Doug,

I'm glad you have worked it out with your clients that rank checks are in-significant. That is not the case with our clients. Have a great day.

Laura Thieme
President
Bizresearch

For those of you who do ranking checks, in addition to other valuable forms of measurement such as traffic, sales/leads and profitability, have any of you ever approached Google about opening up their APIs for a quota fee service, on organic rankings? We've not had much success with Google on this topic, as AJAX and SOAP APIs do not allow you to do much commercially. Considering that Google Adwords API does allow a fee-based quota service to pull all of their data, I'm not sure why we don't have the fee-based ability to pull organic API data, as well as Google Analytics traffic data through an API.

Probably because Google does not see the value in rank checks either.... that's my educated guess. In fact; many of us do not. Educating clients is something that is lacking in this industry in a big way. I'm wondering why it is that some SEO firms know there is no value in rank checking and others think there is? I just saw a post in that spin thread from a tony wright someone. My goodness Tony; you could get the exact same info you are seeking from your stats program. I'm still at a big loss about this stuff as I must be missing something here?

The very best thing Google could do is entirely wipe out the ability of rank robots to scour their databases... and for good. In fact; start enforcing penalties of some kind. Sorry; I just feel strongly about this and really wish education was the main priority in this industry instead of simply going along with whatever a client wants. I hope my doc or lawyer or plumber or electrician or whoever else I need does not go along with what I say when I try to tell them how to do their jobs. I certainly don't want a client telling me how to do my job either. A client hired "you" for the purpose of helping them. If that client has such influence over SEO firms in that they tell the firm they WANT rank check reports..... the SEO industry is doomed. Well, I think it already is anyway.

I have to wonder if Doug Heil actually does any SEO work for clients, given the amount of time he clearly spends writing his bombastic diatribes...Get a life mate.

Thanks Chris. You don't see me "blogging" any crap though. The truth hurts mate. ... it especially hurts you. And no bud; NO clients in ten years. And no bud; I don't do so-called SEO. I haven't for at least 4 years. At least not in the way most do. I certainly don't check silly ranks like YOU do. :-)

Look, I see it all as a matter of opinion. There is no one way to perform effective SEO.

Doug, if you don't want to use ranking reports, then don't. That's fine. No one is on here criticizing you for how you do things. All I would ask of you is that you quit treating your colleagues as if they're less knowledgeable than you simply because you have an opinion.

I use ranking reports simply so my clients who know nothing about SEO can see the steady improvements over time. It's a reassurance tool, a way for me to easily display the work I've performed (aside from the obvious increased conversions).

As far as educating my clients - why would I? Don't get me wrong, I always explain my process and how this will affect that, etc., but ultimately it is in my company's best interest to make sure the clients continue to need my services so they can keep writing the checks. If we teach everyone how to SEO on their own, then we'll all be out of work.

And before you go off on how I should be manually searching terms, all my client sites rank for over 200 keywords each, so a program like WPG is a time saver. Time which my clients pay for.

To each his own, Doug, to each his own...

Wow. No other words can describe.

Justin; yes indeed; to each his own. It's quite unfortunate this SEO industry has many that think just like you do. .. from your very own words even.... sad. Just sad.

And yes indeed; For some reason you think being more knowledgeable than let's say "you" is a bad thing?

You wrote:
"but ultimately it is in my company's best interest to make sure the clients continue to need my services so they can keep writing the checks. If we teach everyone how to SEO on their own, then we'll all be out of work."

Yeah boy; that's what the industry is all about.

Justin; I guess if you are explaining to your clients that running automated rank check reports has been against the Google TOS for many years, then you are covered if or when Google catches you running them for your client's website, right?

Just wondering out loud; but wouldn't it be cheaper if that client bought the auto check software and ran the reports themselves? I think the going price is something like 149 bucks or so, right? Some are even free. You wouldn't have your time involved with actually running the reports, so you could be doing other things maybe. I remember back in 99 running reports. I doubt they have changed much since then. From what I remember, anyone off the street can start and stop the reports.

Jeez - just give us all a break Doug!

You've made your point (ad nauseam); demonstrated you can't accept others having an alternative point of view; & now are being a bombastic bore.

You must be a riot at dinner parties...

Jeez - just give us all a break Doug!

You've made your point (ad nauseam); demonstrated you can't accept others having an alternative point of view; & now are being a bombastic bore.

You must be a riot at dinner parties...

Why do you keep going then Chris? Why can't you accept my opinion as well? Is it because my point is a good one and those who run these reports don't have a good point? Tell you what Chris; you can have the last word as I'm done. Go ahead; call me something else this time. Name calling with "bombastic bore" is weak.

You've missed the point entirely - it's you who won't accept alternative opinions, as you've amply displayed in your numerous posts here & elsewhere.

I think most people would agree that refering to you as a 'bombastic bore' is a statement of fact rather than name calling.

Over & out.

Hi Barry and everyone else...

This is Scott from WebPosition. Google is not blocking WP. If you are being blocked by Google, it is your IP address that gets blocked. If that occurs, you will not be able to search against Google in your web browser either.

Essentially all programs that try to rank check against Google, etc. have to be programmed to find waypoints in the HTML. If those change, i.e. the HTML that is output changes, then programs like WebPosition, WebCEO, etc. may have problems checking rankings. We may not have the same problems at the same time either. We do not necessarily use the same waypoints or means of analyzing the search results.

In no way does this imply that Google is "blocking" or "banning" such software. They simply do not recommend using such software. No where on Google's own site have they published that they have "banned" or "blocked" such software.

Scott,

Thanks for your comment....

(1) You are saying that WebPosition Gold is indeed working properly and users of the software have not had any issues checking their rankings with it over the past several days?

(2) You are saying that Google doesn't say that using rank checking software, such as WebPosition Gold, is against their terms of service?

Really?

Poor Scott! The powers to be at WPG sent him on a mission to bury themselves further. We had an interesting exchange at WebmasterWorld not long ago where Scott joined us.

http://www.webmasterworld.com/google/3716136.htm

70 messages so far.

Think its best not to post anything in public yet scott.. Or at least until you have the full picture..

Hi Barry,
No. I'm saying that Google has changed the HTML that they serve. WP needs to be reprogrammed to understand that new code. Nothing mind blowing, but still a challenge.

I don't think that Google likes automated software. True enough. But associating one thing... 1) that we have to create a software update with 2) Google's dislike for automated software, and 3) inferring that Google is "blocking" WP, is factually incorrect.

Engines make changes all the time and in turn we create updates for WP. I've even created some of these updates myself...

Posts like your original, cause others to post that WP is "blocked" when in reality, it is not.

Scott,

I would not complain about my post. I reported on the thread and that is what we do. I did not state it as fact, I stated it as a question.

I am sorry you have to do damage control, but please do not blame that I me.

The reality is that Google does not like ranking checking software. You are saying Google did not block you, that they changed the structure of the page and code, so you need to update your software. That is fine and please do let us know when you update your software.

If you like, I can ask Google for an official stance on your particular product. Would you like that?

I think the information given in the Webmaster Guidelines and the TOS is fairly straight-forward:

"Don't use unauthorized computer programs to submit pages, check rankings, etc. Such programs consume computing resources and violate our Terms of Service. Google does not recommend the use of products such as WebPosition Gold™ that send automatic or programmatic queries to Google."
and
"5.3 You agree not to access (or attempt to access) any of the Services by any means other than through the interface that is provided by Google, unless you have been specifically allowed to do so in a separate agreement with Google."

Matt Cutts also commented in a thread Scott started in the Webmaster Help groups at http://groups.google.com/group/Google_Webmaster_Help-chit-chat/browse_thread/thread/968ab54325c33870

Hi Barry,

I'm not here on any official damage control mission or what not. A few WP users said, hey Google's down and coincidentally, these guys over here say that Google is "blocking" WP. Is this true? So... that's why I'm putting my 2 cents in.

Hi John,
Thanks for the comment. As Matt posted there: "I would approach the bizdev folks at Google about how to send automated queries to Google with permission." Still waiting for a reply from Bizdev.

Even so, this doesn't answer part of my question...

Why point out WP when there are so many competitors that Google already has the same issue with?

I guarantee you that for every WP user or "automated tool user" who Google thinks is "over doing it", there are others who Google could care less about. I have users that are like "Hey, if I check these 5 keywords once a month, seeing if I'm in the top 10 rankings, am I going to get banned from Google?" No way. Google won't enough sneeze at that.

A neutral, even handed approach would be to say that G doesn't like automated programs etc. but not name any specific program, service, etc. They're pretty neutral with ~other~ policies.

A bias pops up by naming WP and no other program, tools, etc. when there are so many others out there. As Matt also said in that post:
"We do turn off a number of tools/bots/IP addresses that scrape us too
heavily....

...In fact, I know that just a week or so ago our algorithms turned off an IP belonging to one of the entities that you mentioned in your post."

Thus... regardless of your using a tool like WP or not, for Google, this problem is not -just- a WP issue. Google turned off an IP address because of this issue, yet Matt was clear to steer away from naming that company, let alone putting them in the Google guidelines page. In a video post, on another site, Matt was saying that some guy was using a tool, to scrape Page Rank (not WP mind you, we don't scrape Page Rank) so heavily, that they started feeding him bad data. Hrm. I don't see that company or tool's name anywhere on Google. Do you see a bias yet?

Does anyone think the industry can survive without ranking reporting?

Hi Scott, Allow me to interject a thought. Why not change your software so that it does not scrape engines and use their resources? Why not use your software for something that is actually of some benefit to users of it, like, erm, hmm, analyzing log files and web stats? Why should users of your product have to go against Google's clear stance on this issue just to use your product that really does not help them see any kind of big picture anyway?

I'm with pageone; I don't really think it's in your best interest to be talking right about now. I do agree with you that Google should name ALL the products that scrape their services that have to do with stuff like ranks that serve no benefit anyway. It would be very good to list you all in a nice and neat way on the TOS page and the webmaster help pages.

Could anyone here name a tool in the market that (legally) uses the Google API for rank check?

I do like to run rank checks regularly, because it helps me to compare my client to the INDUSTRY.

I do not care if a keywords ranks at 6 or 7, but if a competitor suddenly improves majorly, then we can look into that to see what has been changed... Very helpful in my opinion...

Hi Scott,
I can't help but point out that is Google themselves that name WebPosition Gold.
"Don't use unauthorized computer programs to submit pages, check rankings, etc. Such programs consume computing resources and violate our Terms of Service. Google does not recommend the use of products such as WebPosition Gold™ that send automatic or programmatic queries to Google."

This post was done as a question. You answered it by saying that it wasn't being blocked. Case closed. However it is worth pointing out that Google are obviously doing their best to make it very difficult for WP to run. By doing your updates are indeed finding new ways of breaking that Term of Service. Argue that if you like, but Google quite simply have said that you break their terms.

But agreed that they aren't worried about a handful of end users, it's the power users they are doing their best to block.

lmao...Doug, please understand that many of us reverse engineer the algo by looking at ranking reports to see changes over time. Take a look at some of the advanced SEO materials on the net if you are interested in taking your SEO efforts to the next level :)

Hi Phil, "reverse engineer"; gosh; such big words you are using. Little o'l me is not knowledgeable enough to decipher those type of words used by the "experts" who use these "advanced SEO" thingAmaGiggy technique thangs. Please write in ways I can understand Phil. :D

Well Phil, considering I don't really care about any algo changes that Google could possibly have, I have no need to reverse engineer anything at all.... "never" have. That said; if I ever did feel that need; the log files or a stats program does the same exact thing nicely..... and better.

And guess what? I don't have to violate the TOS of any search engine doing it either. How's them apples? Red I hope. :)

Scott,
Any update on when WP will work again?

Okay so it's been a couple of weeks since I weighed in on this. Paid search KPIs, and traffic/analytics KPIs are to serve a purpose - a metric against which you can improve something. Marketers deliver data on paid search KPIs as well as traffic stats and are measured by whether or not they improve any or all of these metrics. If a conversion rate on a popular ecommerce site is only 2.5%, a company will look to hire someone to improve that conversion rate within time. If your paid search KPI reports show that traffic is down, or conversions are down, and then you note that your average position has slipped from 2 to a 4.5 due to Q score, and your competitors are offering a deal in their ad you can't compete with - then you look for other ways to show up for that term, or you cave to your competitors. If you slip in position, whether it's organic or paid, you can lose business, plain and simple.

What we really need is for a business model that understands the value of paid search, traffic and organic KPI reporting. Google and other search engines could make a revenue case out of companies having access to organic KPI data, as it is just one way that a company can earn traffic that equates to revenue. Google provides this data through a revenue model paid search KPI. Many KPIs cost money, but also generate revenue for the search services. Thus, Google could create a business model here and keep services from scraping HTML code. No one, I guarantee you, wants to scrape as a service. We'd rather pull from an API data service anyday, provided the API data is accurate and reasonably priced.

In closing, I wish Google would make this easier on us. We get access to Paid Search API with an affordable quota price - now we just want a Traffic API, and an Organic API. We'd gladly pay for all, with similar pricing to the Paid Search API.

Laura Thieme, President
Bizresearch

One clarification - Google does not do this yet for Organic KPIs - we're suggesting they do - for a API service fee.

Laura,

I couldn't agree more. I would pay any day for API access to search rank results rather than scrape through WebPosition or any other means.

To Doug Heil:
Yes, we can search for a keyword and check it's rank manually any time, but what if you are tracking several hundred keywords? Don't you find it useful to match ranking data with web analytics data? Isn't it useful to track how your activities are helping or hindering your rankings?

Of course rank reports are a tiny part of real SEO services, but they are an important part if you know how to use them.

Is it just me, or is Doug sincerely the most ignorant contributor in existence? And that ignorance has clearly affected your ability to provide a service to your clients. Good lad.