For many many cases, when people are having issues with their site's doing well in the search engines - they sometimes overlook the obvious. I cannot tell you how many times I have seen or heard that webmasters are complaining their sites are not in the search indexes, but at the same time, their robots.txt command or meta tags are telling the search engine not to crawl.
For example, a Google Webmaster Help thread shows this exact issue.
Webmaster complains they are not in the index, "This seems to be such an area of voodoo for me - so frustrating!, said this webmaster. In response, a webmaster helper, John, said, "You've got: <meta name='robots' content='noindex,nofollow' />."
The webmaster's response, is all to common:
Oh my gosh! I totally forgot about that option in WordPress - thanks for finding it!
Don't overlook the obvious.
Forum discussion at Google Webmaster Help.
