June 4, 2007 Archives

Satya Nadella is a 15 year Microsoft veteran, and has recently become the VP of Micorsoft’s Search & Advertising Platform Group.

There is a new secret search stealth project going on in San Jose?

At the end of the day, we have crack teams in Redmond, SV and China. Lots of other locations and we’re working hard on lots of innovation. When it becomes a reality you’ll hear about it. We’re sure proud of our team in SV.

Is the team charged with doing something different than Windows Live?

No, they’re just part of the team. They’re definitely in the pipeline of all the research and development we have going on.

Now you’ve been on the new job for like a month and a half, have there been any surprises? What are your impressions?

The last time I was directing on the advertising side was when I working on Link Exchange. I’m still learning, but what’s struck me the most is that I feel like we’re the youngest in the business. There’s a lot of fire and desire on our team to contribute. It’s great to be on the team. I’m excited to back in the space, feeling young, and taking some risks and making some innovations.

Any surprises?

The logic of brining the ad side and the search side is pretty self-evident. The fact that we have a full page that has to be relevant to our users,

What do you see as the biggest challenge you have to deal with in the core search as opposed to the core ad side?

When you’re 10% share in the US. The challenge is how do you grow the share so our advertisers are getting more share. To get those searchers though, you need to have the good search results.

I would love to have 50% of Google’s share. To get there we have to crack the code of engagement. There are lots of users, but the thing we really want to do is crack the code of once we get the users trying us, we want to keep them. The MSN audience is definitely a portal audience. They come to MSN looking for a portal experience.

We’ve done a great job integrating Live with Messenger and Outlook.

The other thing we’ve got the exploit is the Live Search in Microsoft.com. Overall, being really able to get the contextual search all around the Microsoft network is really job #1

How do you see the differentiation between MSN, Google and Yahoo?

One of the things I like to think about is, to be in the search game it takes a lot to continuously innovate. If you look at our platform, our algorithms, etc, I feel that we’ve finally reached a level of maturity where we can compete with the best of the best.

There is a lot of innovation in core infrastructure that I feel good about. So next, how do we differentiate? We need to recognize that there is a whole page beyond the ten links. We have answers, related searches… beyond full page relevance is doing stuff with new types of content… images, scratch pad… We quickly responded to Google’s introduction of the face technology on Google Images.

We have now over 120 cities in our 3D web. It’s great for local navigation, and as a new advertising platform. Mobile is a pretty rich app. There’s a variety of these both horizontal and vertical innovations along with our investment in core infrastructure improvements.

Do you think the vertical is going to be the stronger place that pulls ppl in?

When you look at our 10% share, we’re going to try very hard in some of our verticals to improve that. I don’t think we’re in the search game to just get our “fair share”.

We got AdCenter out last year. We’ve been out on the market for about a year. The thing we want to focus on is our quality and the usability of it. Our last update really addressed those last two core issues. Users can do bulk campaign management, etc. A lot of emphasis on basic usability. Beyond that, we’ve started doing contextual content network ads. We’re slowly growing the number of advertisers on it. The CTRs are basically the same as search right now. The traffic that our advertisers expect has been there.

Beyond that, if you go to AdLabs.Microsoft.com you’ll see a lot of the cool stuff we’re working on. Keyword forecasting, and other tools we’re developing for the SEM people.

Another thing we want to do is support new ad types. We’ll be setting the pace on how to innovate and work with the customers to develop new products and ad types.

How can you own an interactive agency and run a search engine?

We intend to make no changes to that arrangement. In some ways, Aquantiv has cracked the code on retaining the separation between the technology and agency, so we’ll be following that pattern and plan to keep both.

The difference is that while Aquantiv owned the tools, they didn’t own the media too. But it’s not the same situation. It’s like the NYTs is running its own PR Company.

It all comes down to what are the policies that govern the functioning of AA? We fully intend to make sure that AA has the flexibility to service their clients in a neutral way. We won’t break the core AA business. Their clients can be assured that they’ll have all the right policies to protect the customer’s interests.

How are you and Steve interacting?

Steve owns the P&L and runs the business. I really have the responsibility to run some of the engineering that runs the P&L that Steve owns. We have a simplicity in how we operate, and it’s been very refreshing to me. In the end of the day, he runs the business; I run some of the aspects of the engineering.

You’ve talked about MSN and Windows Live. Any more changes on that front?

Overall, at the end of the day, MSN is our portal and that’s where most of our traffic and customer engagement is. As far as our destiny in where we make progress with search depends on MSN. We want to use MSN as a brand, as well as a destination.

How do you prepare yourself to “Make Search Better”?

As you change from one area to another… the beauty of MS is that I’ve worked in every part of our business… so there is always the thrill of being in a new domain. I was petrified last night and got very little sleep. I’m sure come Q&A time all my fears will come true. But nevertheless, the best way to energize yourself is to jump into something new. In the morning when I’m running I listen to a data mining class. It’s great to meet a new set of people and learn what they know. We have very, very smart and capable folks, and my job is to be able to take on friction and enable them to do their best work.

Q&A

AARF client – I’ve been using MSN Live Search for the last 18 months. What are your next aims for the next year and where do you see it going?

Answer -- On the search side we want to be able to show some innovation that let’s us improve engagement. How do we take the searchers that are already using MSN Live search and have them use it more. We want to be very scientific and data driven about doing that. All of our work in the verticals will be to increase engagement. You’ll see us do a lot more experimentation in MSN. We’ll do a lot more of these contextual search innovations, especially with Hotmail, Office, and non MSN Live properties. That’s kind of where I see us going.

Larry, Seattle 24x7 – You’ve offered your corporate clients a financial incentive to use MSN Search, will that ever be offered to regular users like in the form of discounts on Vista, etc. Also, what happened to Amazon A9’s street view?

Answer – As to the first one, it’s a good idea. I’ll take it back to my colleagues and see what they say. As to the second, we have street level navigation, especially in Seattle and one other city. We believe it’s a fairly unique property and plan to continue developing it.

(Danny asks about other SE’s versions and privacy issues)

Answer -- Any time you have a situation with user privacy, you put the control of the amount of privacy in the control of the user and let them set the level of exposure.

Danny -- Do you see a way to try to do control over the street level views before the users notify you they want it taken down?

Answer -- This is one of my nightmares coming true  At the end of the day, if the benefit of having the imagery out there outweighs the privacy concerns, then the technology will win out. If it doesn’t, there will be some level of balancing.

Jay, DomainTools.com – Microsoft’s take on Google Universal Search

Answer – We’ve introduced Answers, News, Stocks, etc. I think it’s the same direction Google is going with Universal Search. If you look at the SERP real estate right now, there is a lot more than just the 10 blue links. Directionally, I believe in the direction we’re going in.

Jay, DomainTools.com – Are you guys committed to being compatible with FF?

Answer – Good piece of feedback. We only have 10% of the share, and we definitely want to be where our audience is so we can increase that. There are a lot of areas that we need to ensure we get adequate users.

Danny asks how FF users (all hands)… how many IE users? (no hands)

Danny – I’d almost like to see you “forget” you have IE and develop a product that works independent of it.

Question – To influence ppl to spend more money on the ads on AdCenter, do you think maybe you’d give them a reward via higher natural rankings?

Answer – We’d like to be able to innovate to get the search traffic, and the end user is voting, so we have to make sure we’re providing the highest quality search results.

Question – What do I tell my clients when they ask why they should use Live or MSN?

Answer – Go back to all the things I said about differentiation. We have an image search, a 3D search, and a product search that I believe are all differentiated. There are already 55 million searchers who are using MSN or Live each month. That is the kind of reason why you want to be in multiple SEs and marketing to all the searchers and getting that cross channel exposure.

Danny – Is there anything particularly fascinating you’ve discovered since you’ve come back into search?

Answer – The thing that is the most fascinating for me getting back into the space is the continuous nature of how things improve. From an engineering or computer science perspective, the frequency at which we release new stuff. It’s just all very invigorating for me and I hope to be in this space for awhile.

Personalized Search: Fear or Not?

Moderator:
Danny Sullivan, Editor-in-Chief, Search Engine Land

Speakers:
Matt Cutts, Software Engineer, Google, Inc.
Michael Gray, Owner, Atlas Web Service
Gord Hotchkiss, President and CEO, Enquiro
Tim Mayer, VP of Product Management, Yahoo! Search
Danny Sullivan, Editor-in-Chief, Search Engine Land

Danny speaks first. He mentions that Google is moderating the results based on users' searching habits. He shows us a screenshot of personalized results. It's not a dramatic change, he says. Someone on the bottom of the page might have slipped off the top 10. This is sucky if you are the site that dropped out. But you shouldn't think that Google reshaped everything; it can get more dramatic over time.

Personalization Influencers:
- Add to Google button
- Google Personalized Homepage Content
- Google Bookmarks
- Search History (clicks)
- Web History (visits)

Yahoo! MyWeb lets you go save pages or Block people you don't like. These are signals that Yahoo collects to reshape their results.

Ask.com has stuff that is about a year old. They have the ability to save info (put it into folders). It's not working as well as it should but they have rudimentary things that would reshape the results.

The overall summary is that Google is doing the personalization. Yahoo and Ask harvest information but have yet to do the same as Google. Google is likely to be more aggressive with personalization over time to reduce spam and deliver more relevant results.

Next up is Gord Hotchkiss.

Personalization is a new area. We speculated about what SEO and blackhat might look like.

The thing about SEO in pre-personalization is that there are keywords and algorithms and everything revolves around keywords. But in personalization, it revolves around users: social pattern, search history, web history, and current tasks would revolve around this.

It's very difficult for a marketer to look at an individual user. That becomes very granular. We're going to look at buckets of behavior and work around themes. Themes that fall into common user themes are emphasized instead of keywords. Long tail optimization becomes very interesting. Optimizers will look at the long tail a little bit more where personalization may not be an impact right away. Personalization can really drive a much more presentation of universal search results. If you know more about the user, you're more confident in providing different results to the user. Thus, understanding user behavior is vital. Knowing what people are looking for is critical. User-centric development will finally take hold. You would not believe how many sites are not user-centric. This will really push that.
On the black-hat side, emerging "buzz sites" will be an SEO tactic. You can look at the next hot buzz and find out which will build interests.

One of the other cool things about personalization is getting people close to that final click. Personalization would move optimization higher up the funnel. Get people to your site earlier in the phase (purchase/negotiation/research/awareness).
- Site Stickiness
- Build out of content and functionality
- Sites will become "research bases"
- We're going to see more content aggregation.
- Comparison Wizards
- More mash-ups
- Blackhat: widgets and gadgets

Circles of importance:
A handful of sites or specific content will emerge for each "theme." The optimizers will need to identify the circles of importances. These will be inundated for offers for RSS content, widgets, and gadgets. Scraping of content from circles of important sites will also occur as well (blackhat).

User intelligence will become more important.
- Click stream based intelligence tools
- Engines will influence more profiling tools for paid, which will be used by SEO
- Social bookmarking sites will become hotter
- Personalization enables scaable social search
- More use of personas in SEO
- More spyware in watching click behavior for blackhats.

Now, Michael Gray, aka Graywolf, presents.

Unethical SEO tactics: Suggest to all of your clients to sign into personalized search to search for their site so that they can show up in the #1 position. Then you can say "look how great my work is doing!"

Fear of uncertainty: different datacenters and geotargeting already create SERPs with varying degrees of differences. Personalized search adds more uncertainty to the issue. PPC might actually work better in terms of stability.

SO you can become a Google addict. If you want to take advantage of personalized search, you need to use Google Reader, Google Bookmarks, and all other Google properties.

How is Google dumbing down the user? People are assuming that Google knows what they want and will provide results that they want. Then people trust Google too much and use other search engines and services less. I don't recommend that.

How do you fix it? Stop hiding that people are logged in in a very obscure part of the screen. Be clearer when the results are personalized search results and are not normal results. Make it easy for people to turn off personalized search.

Take advantage of social media sites to get more personalized search results.

Tim Mayer is up next. He mentions that he'll discuss how Yahoo looks at personalization and their approach in the social area.

Two techniques of search personalization:
1. Session based personalization - understand the intent of the user based on queries and clicks during a specific session. The challenge of this is figuring out when sessions start/end. You might want to search for the Florida Marlins but then you switch to fishing and search for "marlins" again but might get the wrong results.
2. Interest based personalization - Understand the interests of the user based on their own declared preferences or user behavior inside or outside of the search context. The challenge of this is that sometimes users do things outside their normal behavior. If you are searching for Jaguar the car before, but then you got a Mac, your behavior may be not normal.

The impact of personalization on search results:
- Queries should get shorter: current average is ~2.7
- Example: [library] vs. [boston library]
More of the top 10 should be relevant to the user assuming the intent is extrapolated from the query.

Personalization's impact on SEO:
- Better relevancy: better matching results that show up
- Give the search engine enough content per page to help it determine the topicality of that page.

Yahoo's Approach - we take a more social approach (delicious, Flickr, Yahoo! Answers). By searching and leveraging this data, we help people discover new forms of information. Discovery and recovery. You can store this data and find it at a later date.

Many results are subjective - cool lamps on Google vs. cool lamps on Flickr. People's reputations are on the line on sites like Flickr. There's social incentive for people to help you and tag them appropriately.

Finally, Matt Cutts is up last.

Personalized search is different for people in different ways. For example, I can go up to Jane and ask her "What's a Kiwi?" Since she's from New Zealand, she's going to say something different.

One of the coolest things about personalized search was that once, I was looking to make a vignette on Adobe Illustrator and there are like no search results that are relevant for that finding. A few months later, I needed to find this again. How am I supposed to recall the search terms I use? I can look at my Web History and see what I searched for and what I clicked for.

There's a whole new era to optimize in personalized search. SEO won't be the same as before, but personalization is not such a huge change that things will be unrecognizable.

This is how you disable personalized search: with 6 characters, you can add this parameter to the end of your search - &pws=0. It's easy to opt out.

Greg Linden once had commentary on Personalized News Search and said that he has read that there is a 40-50% improvement on clickthroughs. But his commentary says that there was more of a 200% improvement on clickthroughs. And this is just in Google News. It will be great to explore this space in the future.

SEO, Meet SMM

Moderated by Danny Sullivan

Presented by:
Rand Fishkin, SEOmoz
Cindy Krum, Blue Moon Works Inc.
Todd Malicoat, Stuntdubl
Neil Patel, ACS/Pronet Advertising

Danny introduces the session. He mentions that while this is a relatively advanced conference, there is fundamental coverage in this session because search engine optimizers really need to understand social media. Social media traffic is great for a site. A lot of people are going to social media sites and are doing a basic behavior of discovery. They can do a keyword search and browse the headlines or go to Digg and learn that the whole world is about Macintosh. (And it's true.)

First up is Rand Fishkin.

He's going to cover social media marketing vs. viral marketing.

You go to Web 2.0 sites and approach these sites in the desire to be part of the community. We make the Yahoo, MSN, and Ask communities work for us. This now extends to social media to leverage them for success. How does this work?
- You can rule the search results. (You can push down negative content in reputation management cases.)
- You can get link love.
- Traffic,
- You can influence the traditional media as well.

Where can you do this?
1. YouTube
Not directly influential over search results but has millions of views.
2. Wikipedia
3. Yahoo! answers - a phenomenal resource and a great place to participate.
4. Yelp - most popular local reviews on West Coast.
5. LinkedIn - as a personal networking source.
6. Flickr - Those comments don't have a nofollow on them.
7. Craigslist - bestof Craigslist are the best links you can get.
8. Facebook
9. Amazon - blogs, lists, participation.
10. MySpace - SEOmoz ranks.
11. Technorati - you definitely should be tagging your content.
12. Judy's Book - like Yelp, it's a local service.
13. Newsvine - contributing a lot can earn you rankings in search results.
14. Twitter - not useful. [Danny says that it sends you a lot of traffic.]
15. CitySearch - good local directory.
16. WikiHow - great place to put articles

What can viral media marketing do for you?
Do you remember when there was no YouTube? In December of 2005, YouTube became more popular in a month. What did that? Lazy Sunday from Saturday Night Live was put there, and now YouTube dominates the market.
Viral media gives you the ability to brand.
Viral media causes search rankings through links.
Viral media means that you are growing your fan base. RSS subscribers go up.

Where can you submit linkbait?
1. Digg
2. Reddit (politically focused more so, real content focus rather than Apple/Wii fanboys)
3. StumbleUpon - toolbar sends a lot of traffic. There are 2.5 million users click on the Stumble bar when they are on the phone with their clients. (I've done it.)
4. del.icio.us
5. Netscape - a little less valuable but good traffic. More serious news.
6. TechCrunch - my understanding is that unless you can give Arrington something he wants, you won't get on here. But if you do get on here, there's phenomenal traffic.
7. Newsvine
8. Boing Boing
9. Fark
10. Engadget
11. Techmeme
12. Lifehacker
13. Yahoo! Picks - send decent traffic. It's tough to get there.

Is viral marketing really that powerful? Yes.
SEOMoz has 724,200 links since it was launched in 10/04 and has a PR of 7.
etc.

Next up is Neil Patel.

There are tons of social media sites out there that have a bunch of terms. They don't want you to pay for votes, create multiple accounts (especially from the same IP), or submit illegal content (pornographic content, or how to get free DirectTV).

Neil broke the rules and eventually his username was banned.

It's not long term and it's not effective.

Consider your audience. They are arrogant babies with attitude and spunk. (An image of a punk baby is on the overhead projector.) The older audience is really 28, 29, and 30 years old.

Unwritten rules: they don't want you to do self promotion, add biased information, or ask friends for vote. The users will respond by saying "stop spamming your link on all the front page stories" and call you out on it.

Neil then presents his own golden rules:
- Add tons of friends. They don't have to be your real friends. If you submit a piece of a content and tons of people vote on it, that means they're interested in the stuff you're submitting, so become their friends.
- Participate in the community. Get involved with the people who leave their AIM screenname or email in their profile.
- Become a top user. You can submit stuff and get away with it. People look up to you. They will vote on your stuff and you'll succeed.
- Use their features against them. If they give you tools to become friends, become friends with them.
- Create a social brand. Don't use the name like "neilpatel" all over. People might recognize you by your unique name and avatar.

Neil shows an example of the StumbleUpon tool and the ability to send a link to your friends. You force them to visit your page.

If you want to succeed, do what's ethical at the end of the day. Don't do spammy stuff unless you get a lot of traffic and links. Think about your brand and don't jeopardize it. Think long term. Most people think about social media for a quick fix. It's a long term strategy.

Todd Malicoat from Stuntdubl comes up next.

Firstly, the benefits to strategic linking with social media.
Control over anchor text, body copy, theme, and lots of opportunity for links from trusted sources
Rankings, traffic, and sales go up.
He shows an example using Squidoo.

Caveat: invisible nofollows
With Digg, sometimes the links won't follow at all.

Spam is determined by intent and extent. Search engines might see this but engineers won't and they won't last for very long.
Size matters:
- link buying vs. strategic placement
- doorway pages vs. landing pages
- social media spamming vs. social media optimization

Strategies: build reputation neighborhood which is good for reputation management, build hubs (e.g. Squidoo) with social media links, maximize the use of anchor text, and looking for social media sites.

Reputation neighborhoods - hat tip to Michael Gray: sign up to everything you can find, test them and see which rank; if they are good, link to them; build a neighborhood of sites; interlink within reason, etc.

Some of the sites include: Squidoo, Naymz, Netscape, LinkedIn, Tagalag, Bill Hartzer's 130 social media sites post on WMW (paid subscription required).

Building a hub - you can use MySpace - e.g. John Tucker Must Die (MySpace ranks higher than johntuckermustdie.com).

How to find sites: Google dorks, Keywords + "profile", Keywords + "web 2.0", Directories like www.go2web20.net

Test social media sites for link value: find out if the site is indexed. Is the site indexed? Is the page indexed? Do these pages rank? Is the theme good? Placement (how many links). Does the page pass link juice?
Testing methodology: run a site:command on similar pages on the site. Build on the social media site filled with nonsense, and link to another nonsense page. For more, search for "case studies" on Wolf-Howl.com.

Conclusions - Benefits: know how to use the tools. Caveats - size matters - and so does intent and extent. Use strategies and experiment.

Cindy Krum is up last.

She talks about social media for brand awareness.

Why is social media important to your brand? Ubiquitous adoption of web technology means your customers will demand a higher level of interaction with your brand. Social networks have an undeniable power. Radical transparency is the new public relations nirvana.
Social network advertising is expected to triple by 2011.

Secure your social presence:
* Research relevant social networks.
Major social sites.
You can also create your own social sites.
Look for niche/vertical sites.
Blogs and forums.
Wiki sites.
* Identify existing networks for your brand and industry
* Have a strategy
How many profiles will you create, and where should you direct traffic?
* Determine who gets social profiles.
Brand, products, and company icons.

Consider social profile portals to your brand.
You want people to be part of your community and to come back. Keep the information fresh and current. People should want to come back because they see constant updates.
Manage multiple social profiles in centralized locations. Use remote photo hosting instead of hosting on multiple sites.
Leverage email functionality, blogs, and billboards.

Create SEO'd profiles:
- Follow traditional SEO best practices
- Focus on brand keywords
- Interlink profiles to your brand sites
- Initiate friending campaigns
- Drive traffic to the profile (natural search traffic, main site, PPC, PayPerPost, banners, offline)

Manage your reputation: use SEO to push detractors out of top positions; send traffic to positive press; participate in forums and groups about detractors and supporters.

Empower brand evangelists - give them cool stuff:
Widgets, profile layouts, graphics, desktop themes, videos, podcasts, surveys, customizable HTML, promotion codes/special deals, more info about your brand

Starbucks, for example, gives out a widget. You can add MySpace profiles and there's a widget where you can invite people to drink coffee with you. Another one is True (Create a Date), which was a MySpace featured profile. You can play a game where you can manipulate someone's head and stretch out his nose. There are fortune cookie widgets, etc. LOST is another one - staying on top of the buzz about the show.

Embrace convergence - leverage existing marketing efforts. Send TV/radio/print traffic to profile pages. Release, promote, and link to commercials on your profile pages. Link to information/press about the campaign from the profile. Use profile to get feedback.

Direct social network traffic OFFLINE - host meetups offline; offer in-store only coupons, use social profiles to integrate on and offline brand interaction (contests, scavenger hunts, offline games - create communities offline and post results online).
Direct social network traffic ONLINE - create a social media section on your site and link it from your homepage. Encourage visitors to add you as a friend. Encourage participation.
Promote all the cool stuff you're giving away on your profile. Send them to the profile to get it.

Duplicate Content Summit

Moderated by Danny Sullivan
Speakers: Vanessa Fox, Google; Amit Kumar, Yahoo; Peter Linsley, Ask.com; and Eytan Seidman, Microsoft Live

Eytan Seidman of Microsoft is up first.

Why duplicate content matters - on a search engine side and what you should be thinking about as a webmaster.
I'll go over the two types of duplicate content - accidental content that is created, and duplicate content that is taken from you.
I will give principles and then explain how we (Microsoft) handles it.

When you think of duplicate content, it basically fragments your page in some way. You're fragmenting your rank. The page you might want to appear might have different versions of it now.
Let's look at a scenario: http://mycompany.com/seattle/posters.htm. someone then tells you it's better to have the keywords in a subdomain so you create http://seattle-posters.mycompany.com. Without the proper redirects, you might run into duplicate content issues.
One thing you should think about is session parameters - keep them very simple. Try to avoid feeding the engine a page that has a ton of traffic parameters in it. Duplicate content for locations is also something to think about - if you have unique content, it's fine. But if you have duplicate content for the UK and for the US, it's something to think about.
Redirects: as much as possible, always use client-side redirects. Tell the client to redirect rather than do it server-side. JCPenny.com vs. JCPenney.com - it's a duplicate site. Another one is Wikipedia. Sometimes they have similar terms and a server side injection. This is what hurts your ability to concentrate your rank on one page.
Another thing to consider is http vs. https. If parts of your site need to be secure, you shouldn't duplicate it all on https.

How do you avoid having people copy your content? All my experience is based on sites I helped administer. One thing is a simple method - tell people that if they use your content, they should attribute it to you. You can also block out types of crawlers, detect user agents, block unknown IP addresses from crawling.
There's a blog post on bot verification: http://blogs.msdn.com/livesearch/archive/2006/11/29/search-robots-in-disguise.aspx. It's important to differentiate on whether people are maliciously duplicating your content or if they're trying to help you.

If you think you might duplicate content, consider the following:
- Is there any value to duplicate it? Or are you adding new value?
- If you're going to take someone's content, make sure you give attribution.
- If you have "local" pages, block it using robots.txt.

How does Live search handle duplicate content:
We don't have site-wide penalties. We look aggressively for session parameters and tracking parameters at crawl time. We don't want false positives. One way to help us is to redirect those to be hidden from our crawlers. We filter duplicates at run time. We want to give users content that is unique.

The next person who speaks is Peter Linsley from Ask.com.

What's the standard definition of duplicate content? You have the same content on multiple URLs. It's rarely a good idea.

Why is this an issue for search engines? It impairs user experience and it consumes resources.
Why is this an issue for webmasters? Fragmentation of pages - you really want to put everything in one place. You should care because search engines might index the wrong page and you don't want to leave it to us. Some cases are beyond your control (scrapers).
These concerns are valid but are rare.

How does Ask.com handle duplicate content?
It's not a penalty. It's basically the same as not being crawled. It's performed on indexable content - so templates and fluff (footer/header) are not considered.
We only filter when the confidence factor is high. There's a low tolerance on false positives.
A duplicate content candidate is identified from numerous signals - similar to ranking, the most popular is identified.

What can you do?
1- Act on the areas you're in control of.
Consolidate your content under a single URL or implement 301 redirects.
Putting up a copyright or creative commons notice.
Uniquify content.
2- Make it hard for scrapers.
Mark your territory. Try to make your content so that it cannot be used in a generic context.
Take legal action.
3- Contact Us
You can send a re-inclusion request if you suspect you're being filtered out.

Potential questions for the Q&A:
Technical side: webmaster outreach, W3C URL standardization, watermarking/authentication, search engines improve in anti-spam
Legal: what else can be done?
Economic: make it harder to monetize.

The third person who is up is Amit Kumar of Yahoo:

I'm going to concentrate on Yahoo-specific things and explain what we consider okay and what we hope people will do less often.

Where does Yahoo search eliminate dupe?
We try to extract links during the crawl. We're less likely to extract links from pages that we know are duplicates and we're less likely to crawl new documents from duplicate sites. Primarily, we try to keep as many duplicate documents as possible in our index and use query time duplicate elimination: limits of 2 URLs/host and domain restrictions.

There are legitimate reasons to duplicate -
1. Alternate document formats - PDF, printer friendly pages
2. Legitimate syndication (newspaper sites have wire-service stories)
3. Different languages
4. Partial duplicate pages: navigation, common site elements, disclaimers.

Accidental duplication -
Session IDs in URLs: remember, to search engines, a URL is a URL. We can only crawl so much.
Two URLs that refer to the same document look like duplicates. We can sort this out but it may inhibit crawling.
Embedding session IDs in non-dynamic URLs doesn't change the fundamental problem.

The other accident page is Soft 404s - not found error pages should return a 404 code instead of a 200 status code. If not, we can crawl many pages of the "not found" page.

Dodgy duplication -
- Replication content across multiple domains
- Aggregation
- Identical content with minimal value

Abuse:
- Scraper spammers, weaving/stitching, bulk cross-domain duplication, bulk duplication with small changes.
All of these are outside our content guidelines and can lead to unanticipated results for publishers.

How you can help us:
* Avoid bulk duplication of underlying documents. Do search engines need all versions of pages with small variations? Use robots.txt.
* Avoid accidental proliferation of many URLs for the same documents - sessionIDs, 404s, etc. Consider sessionID-free or cookie-free paths for crawlers. They are not abusive for our guidelines but may impair effective crawling.
* Avoid duplication of sites across many domains.
* When importing content from somewhere else:
- Attribution.
- Ask: do you have the rights to it? Are you adding value in addition or just duplicating?

In the last few months, we came out with Robots nocontent. It's a microformat like tags to mark up low-value content (like disclaimers, etc.). It's useful to indicate where the core content is. We also came out with the ability to remove URLs that you don't want to be crawled.

The final person who speaks is Vanessa Fox of Google:

There are a lot of different kinds of duplicate sites.

Similar content: I'm sure you've seen the Buffy the Vampire Slayer episode where they had 2 Xanders. You just need to combine them into one page.

But sometimes they are more similar but different. There are 2 Willows - and these were not the same. One was more evil than the other. They just needed to be distinguished a little more. That is kind of easy - you should know how to do that because you have all watched Buffy.

Other things: syndicated content, manufacturer's database, printable versions, multiple language and countries. You need to add value to distinguish your version of the database but you might want to add stuff to robots.txt for printable versions.

Blogs are more of an issue lately becasue of RSS, archive pages, category pages. There are those issues - scraping types of issues.

What else do you hate? Would you want to have the ability to rewrite a URL and count everything the same for crawlers. Can you verify authorship?

Question and Answer:

Q: I was very surprised when Eytan said that redirects should be client side. How does a bot cope with meta refresh rather than a 301? Can the 4 search engines agree on a variable/parameter to track URLs (to gets around the sessionID problem)?
Eytan: When I say client side, I'm including things like 301 redirect. I do include that in 301 redirect. When I refer to server-side, I refer to injections.
Amit: In the JCPenney example, there are 2 identical sites. Who is copying who? Having a redirect is important for this.
Peter: For the most part, a meta refresh is the same thing as a 301.
Amit: About the parameter, Google had a note in the guideline not to have an id in the URL. But that has been removed. It's hard for us to figure it out. We're all working on making this better for you.
Vanessa: Maybe not all CMSes can handle that. Potentially, you can tell us what the parameter is and we can do a rewrite.
Danny: How many of you want to express that through a robots.txt file?
Someone says "as many [ways] as possible."

Q: On nofollows, if we use too many of those to get rid of duplicate content, would that be like a red flag?
Vanessa: You really only have control over your link. People can still link to your page without a nofollow. You might want to robots.txt them out instead.

Q: I work with a lot of companies that use Wordpress as a CMS. There are multiple author designations, etc. There is a concern that these are duplicate content. They are, however, but I want them to be referencable search result for at least 2-3 different tags. Can you reach out and work with blogging platforms so that we don't have to do it ourselves?
Vanessa: We should do that more so it's easier for a site owner.
Eytan: What's the easiest solution you're trying to accomplish?
Followup: When someone blogs, how do you designate the primary page vs. a tag page?
Vanessa: We can usually sort that stuff out.
Amit: It goes back to - is there a specific thing that you're looking for: "here's what I expected, here's what I found?"

Danny: there are 3 ways for duplicate content - scraping, syndication, and site-specific duplicate content (you have 2 versions of an English language - like UK vs. US).
He asks for a show of hands and most people are concerned about duplicate content within their own site.

Q: I have a question about daystamp, timestamp, and the date that the page has been discovered. I've seen people's sites being indexed but they were discovered before mine. How do you figure out who is first?
Eytan: Over time, outside of the news scope, it's not really a big part of it. Over time, we're looking for other signals to determine which is the canonical source for the content. There definitely will be an aspect of that. We do leverage for both scenarios.
Peter: It's gameable so we don't act so much on it.
Amit: You can use a sitemap to help us determine who created it first.
Danny: When you try to figure out what is the best copy, it's usually the "first copy." But some people might consider it the most-linked-to copy. What is it?
Eytan: Usually, it's what ranks best in run-time. We're not looking at time. We're looking at a huge number of other factor. If someone copied TechCrunch or Search Engine Land verbatim, they probably won't be able to rank higher.
Danny: What if you said "this is the date of this document to prove that I got there first?"
Peter: It's something to think about but it is gamable so we need to be careful.
Danny: There ought to be a way for search engines to see that it came from a sitemap because the search engine was pinged from this site first.

Q: My question is regarding sitemaps. A few months ago, something was announced about unifying the standard and we haven't heard anything about it since. Is there continued work on this and who is involved?
Danny: They unified. They walked hand in hand. It was beautiful.
Vanessa: Yes. I believe everyone is supporting it. We meet regularly to discuss other things that we can do. I would watch for more stuff because we have more stuff planned as we evolved.
Amit: We created this format to get a long list of URLs. You should expect a lot in the future.
Eytan: We're experimenting with auto-discovery in robots.txt. You should see stuff within 6 months or so.

Q: Tracking URLs and parameters are my biggest concern.
Vanessa: I think we need to link to the canonical version to prevent dilution.

Q: We have a site that has data in the forms of graphs instead of text. How do you recommend that a search engine knows that this is unique content?
Vanessa: Can you have a textual description for graphs?
Followup: We have done that.
Peter: Taking that a step further, use that description as the title of the page.
Vanessa: I'm assuming your graph is an image, so we're not going to assume that they are duplicate pages.
Eytan: Images don't really get parsed right. We don't know what the graphs mean. Adding text that differentiates the page would help a lot.

Q: I have a site that does how-to videos. Should we be worried about duplicate content for those videos?
Vanessa: You're thinking for video search results?
Followup: Exactly.
Amit: Does the syndication of the video point back to the original page?
Followup: In some cases, yes, in some cases, no. We do upload to YouTube and it goes back to your site.
Vanessa: You can block with Robots.txt. I don't know too much about video search. I can try to find out more.

Q: I have 2 suggestions/questions. First is for digital signatures to prevent scraping - can we put in a unique identifying code to prove that we're the source? Also, there are no good reporting tools that indicate what the engines consider as duplicate content. What do you suggest?
Danny: I think that would be cool. How many people would like that? [Applause.]
Going back to the watermark thing, it's really hard because it needs to be a standardized format.
Eytan: We've seen these tried these in other mediums like email with fairly moderate success. How do you get people to adopt this?
Michael Gray shouts from the audience: If you don't offer it, nobody can adopt it.
Vanessa: If the original person doesn't authenticate but a more savvy person does, they can claim your content.
Amit: We'll certainly look at all these things.
Danny: I do have to give them credit. They rolled out a great amount of things in the past few years. Some stuff works, some stuff doesn't, but they try. Duplicate content reporting would be cool.

Q: Danny, you raised a question of what was the biggest problem and some people said that their content was scraped. We have resellers and we give them our content. They've used our content. Hundreds of thousands of sites have our content. What do we do now short of rewriting our site?
Danny: That's where a duplicate content reporting tool would be helpful. How do you prove that it's your content though? That's where it gets difficult. You can possibly do manual reviews.
Followup: What do I do right now?
Vanessa: In your situation, it may be harder for you to contact all these people. Your best option really is to rewrite your content and make your content better.
Followup: We gave the content out beforehand. It was great before, but it's a real problem for us now.
Amit: In perspective, a reseller who has your content - this affects you if a page is exactly identical to your page. It's very hard given all the factors if that's why your page isn't being indexed. There are other reasons why your page may not be indexed, like your reseller is more established than you are. Then you might have to think about how to build your brand rather than focus on duplicate issues. I'd be a little careful about attributing to that source of duplicate content.

Q: A little different duplicate content problem - sites like eBay that have good SEO teams - you get multiple listings from the same business. eBay's subdomains are blatant and take over all the results.
Vanessa: From our perspective, we want to show a variety of results from a variety of sites. We'll look at this.
Followup: For years, eBay has done this and not a single search engine has addressed it.
Danny: Google hates eBay now and Yahoo likes them. [laughter]
Amit: I know you're being facetious but it doesn't matter. Give us test-cases. Show us which hosts from eBay shouldn't appear. We're always looking for these test-cases. Please submit them. We're absolutely working on it. We don't want other companies with better SEO expertise any higher priority.
Eytan: It's a ranking challenge as well. Perhaps the best results are on eBay's properties. There are filters at runtime. We all want to show the user content that is substantially unique.

Q: I want to question the premise of this panel. Why should we only see one listing of a piece of content? Let's say I wanted to find a fact - Abraham Lincoln's birthday. If Wikipedia comes up, what if I don't want Wikipedia. What about a way to group together duplicates?
Danny: Do it in AJAX. It's all nice and slick.
Peter: When we talk about duplicates, we're talking about the same exact content.
Vanessa: Your example - if you don't want a site like Wikipedia - you might find a site that has the same stuff as Wikipedia.
Followup: In the retail area, is it important to have 5000 descriptions of red widget #5?
Vanessa: That's something we can take a look at for the user experience side. We might want to do experiments to see what would be the best and have that option available.

You&A with Matt Cutts

At first, Matt comes in and says that Danny, who introduced him to the session, is not dressed casual enough. Danny, therefore, strips into a Disneyland shirt, cargo shorts, and funny shoes.

Michael Martinez asks the first question. A lot of people have found that supplemental pages don't come up. Is this the way that they'll be in the future, or will they be found in search results if there are links in the near future?
Matt: Both of these are true. If you get more links in supplemental results, they can get in the main index. We parse pages differently that are in the supplemental index (than in regular SERPs). That's what distinguishes the main index from the supplemental index; supplemental index pages are compressed more often. The phrase relationships are handled a bit differently. You shouldn't worry too much about how many pages you have in the supplemental index. I have hundreds of pages in there; that's fine.

Question: You're doing more and more to figure out where the user is to deliver relevant results. What percentage of your results do you feel are completely relevant?
Matt: That's a little outside my area of expertise. We can target people geographically.

Question: Pat from feedthebot asks about the Google Guidelines. I'm curious to see where you see the future of the Google Guidelines and why are they so undescriptive that I have to write about them?
Matt: People should be able to think. It would be really good to have examples and the folks, like Vanessa, who are on the Google Webmaster Central, are working on making this clearer. Having more details is a pretty good way to go forward.
Pat: Feel free to use feedthebot.

Question: Speaking of link schemes, I came across an ad looking for a SEO manager and they were looking for "expertise in buying links." What would you say about this?
Matt: When people advertise for that, you have no idea how many people write to Google and say they are offended. We get those job descriptions forwarded to us. I wouldn't be surprised that we have something like this in the Webmaster Console in the future, just like we had the Spam Report form (which we had since November of 2001). We try to approach things algorithmically and also take people into account. We consider buying links to be outside of our guidelines and we might take strong actions on that in the future. If people want to ignore that, we as a search engine might take action on that because we want a high-quality index.

Question: Since we're talking about links, the more links you get, the higher you get ranked. What about links that come out of your site?
Matt: I think it's good for your users, and therefore it's good for search engines.

Question: I want to talk about search pages and indexing of search pages. We have a catalog of many items and we want these indexed. How can you make it user friendly and search engine friendly?
Matt: We basically said that you should avoid search result pages for good results for your users. However, that's not in the quality/spam guidelines; that's in the technical guidelines. It's more of a "best practices" guideline. We do reserve the right to remove result pages that don't add value to the user. What is the value at? If it just looks like search results that are available anywhere else, you have a problem. If you have unique content, that's much better. Categories are great. Pretend you're a competitor and ask "would this be a good page?" It will help you think of search in new ways.

Question: What is the impact of click-through on authoritativeness (personalized search aside) in organic results?
Matt: We haven't talked about whether it will affect general web search. What I will say is that if you were to use that as a signal, it would be very noisy. If usage metrics are involved, everyone will jump onto it. I'm afraid of using metrics like that. I think MSN has actually said, "yeah, we use that." But because of the people who would try to optimize, we haven't talked much about those signals and to which degree.

Question: Why does Google love Wikipedia? When will you break up with him?
[Everybody laughs.]
Matt: That's an interesting question. Let me put it to you differently. By definition, people who go to Google are not regular users. [Added by Barry: Matt said those that are attending the SMX conference are not like normal Google users. I.e. SEOs and SEMs are not the average Google user.] This question actually came up on SEOmoz where they asked "why do you rank Wikipedia over accurate sites?" Regular users do like Wikipedia a lot. That said, it's not always the right answer. It's a fairly good result most of the time, but for expert results, sometimes it's not the most accurate. We change the algorithms to make the most accurate result #1.
Followup: I work for Edmunds.com and Wikipedia bounced our results down.
Matt: I'll take that into account.

Question: At Pubcon, the first thing you did was look up every domain everybody owns. So I have a question - what business is that of yours, and can other domains affect this?
Matt: If you have 10 sites and 6 of them are catalog sites and 4 of them are hardcore porn, that doesn't bother me. But if a webmaster has two sites versus 50 sites, there's something to talk about. That's a different webmaster than the one who has 2 sites. My goal was to determine if this guy was a power webmaster or if he's a beginner webmaster.
Followup: Algorithmically, it has nothing to do with it.
Matt: [silence]
[Laughter]
Matt: I would consider that fair game. Another site by itself though might not be bad. But if you have 200 sites that we think are spammy, that may not be the best.

Question: Jason Calacanis has said that Yahoo and Google have too much spam in their results and he thinks that his results in Mahalo have more relevant results.
Matt: It's too early to say because his site was just launched. We have all these other sites - chacha, Sproose - I support these. Let a thousand flowers bloom; let's try these approaches. The whole idea is that humans have nothing to do with search at Google is not true. PageRank is based on hyperlinks. We have toolbar voting, we have report spam - we look at our webmaster guidelines and try to remove the wording focused more on "algorithmic" results to allow for possibilities to include more human involvement. It's good if Google changes things for things like social search. We've used humans in many different ways.

Question: I have a followup on categorization and permutations. We have millions of products and our category pages are in trouble. How do I put this against my competitors?
Matt: Suppose your product is in 3 categories - shoes, sneakers, "best" category. Find the best-applicable category instead of having the same object show up 30 different times. That's why you're outside bounds.
Followup: But if you have color, shape, etc - you'll still have trouble.
Matt: It's difficult to slice and dice that. Color as a category may not be as important. Take a look at your category - talk to your users, and only work with the categories that are most important. As far as competing with your resellers, that's tough. A lot of these companies are savvy - look at what they're doing. You can learn a lot from competitor analysis.

Question: I wanted to follow up on the algorithmic vs. human powered in terms of a recent Google Bomb that was diffused. Everyone seemed to agree that it was not a human intervention, but now you're saying that there is human intervention. Please clarify.
Matt: The Googlebomb algorithm is completely algorithmic. My understanding is that the algorithm was not changed. It doesn't run every day. It runs every 2/3/4 months. I think Google does reserve the right to use humans in a scalable way and to take manual action on spam, and to the best of my knowledge, it has remained that way.
Danny asks: What about Bush and failure? Did you keep pushing the button?
Matt: By putting the word on the page, it comes up as a valid match. There is a thing that determines potential Google Bombs, but in that instance, they were engaging in discussions ("failure" was on the page).

Question: I have a question about image results in the search results page. How is this going to evolve?
Matt: We always have a Onebox. When people look for "sunset," they are often looking for an image of a sunset. That's what our Onebox did. With Universal Search, we're looking for the right result. If you're looking for "fix a sink," it might be appropriate to have a video result. We'll check out the ones you mentioned ("George Bush" shows Jimmy Carter image; "retarded" shows people's photoshopped image). We're getting better at image analysis but by no means are we perfect. People can come and say "the search for my name is incorrect" and we'll try to fix it.

Question: A few years ago there was a University of Washington video where they put similar keywords in results (LSI, themes). How has that technology evolved? What about a theme across a website -- people think that it will dilute results. Is that true?
Matt: My bottom answer is "try it and see what works for you." You might have "bio," "biography," "discography," etc. Work these in a very organic way. You don't need to do it artificially. Work these synonyms in a natural way. There are people who are fans of LSI. Google does a lot of work behind the scenes to do good semantic matching. If people are searching for bios, they are generally looking for biographies. But Apple and apples is not the same thing, for example. If you can do it on your own, that's fantastic, but we as Google will try to do semantic understanding ourselves under the hood.

Matt asks: What do you guys want from the Webmaster Console?
People answer: Penalty reports, real time information, accurate reports, errors without having to go into each domain, spider traps, shared logins, RSS, 404 referrers, more data on a query.

After missing the shuttle to the conference center from the hotel, I found two folks to share a cab with. A fellow from Shopping.com and a nice lady who is reported for Loren at Search Engine Journal.

I went through registration, bumped into Tamar and grabbed a soda. I then sat next to Chris Sherman and we fought over the free wifi.

Then as we sat, the music selection before the first session started, seriously rocked. Really good music.

Then they introduced Danny Sullivan, the host of the Search Marketing Expo conference. It was an exciting greeting.

Danny then comes up to the stage, with a headset microphone. He gives the introduction and so forth. Basically, thanking sponsors, explaining how this conference will differ, thanking everyone for coming and finally going over the sessions and events.

Danny also thanked Mike Grehan for giving him the name, SearchMarketingExpo.com. He then goes into the other shows out there, more details at http://searchmarketingexpo.com/.

That is all from me, Tamar is going to take over from here and cover the rest of this session with Matt Cutts and Danny. Boy am I tired.

On Friday afternoon, I wrote at Search Engine Land that Google Confirms Acquisition Of FeedBurner. I have my conference call notes of Susan Wojcicki, VP Product Management of Google, and Dick Costolo, co-founder and CEO of FeedBurner answering questions from the press. I tried my best to get answers to more marketing oriented questions but most of the questions and answers were financially focused (as you would imagine).

I wanted to first congratulate the FeedBurner team. I have been a huge fan of their services from day one and I am happy for them. As much as I was happy for Urchin, when Google bought them.

Now, let's go to the forums, to see how SEOs and SEMs feel about the news.

There is forum discussion at both Search Engine Watch Forums and DigitalPoint Forums and here are some highlighted posts:

It's a reasonable assessment that Google's acquisition of Feedburner simply increases their possible ad inventory (as well as offering some very juicy data). I think that it's highly unlikely that AdWords would be forced onto feeds though, especially without re-numeration. I can see a blogger style implementation happening, where they make it very easy (or maybe the default) to use AdSense in the feeds.

If it means we could be seeing an easier entry into their feed advertising programmes, I'm happy. Feedburner has (had?) a very high requirement like 500 subscribers (unsubstantiated; read somewhere).

BOTTOM LINE: This acquisition increases reach and provides new opportunities for both AdWords advertisers and AdSense publishers. At the same time, it helps Google provide even better results in the SERPS.

I agree that there is outstanding synergy between FeedBurner and Google. I am pretty excited to see how this will help the publishers, the advertisers and those who love to look at stats.

Forum discussion at Search Engine Watch Forums and DigitalPoint Forums and WebmasterWorld.

Everyone is talking about the Google Keeps Tweaking Its Search Engine article from the New York Times. Check out the Techmeme coverage and you can find dozens of summaries of the article.

I will summarize the article based on the three different forum threads I found.

WebmasterWorld has several comments including:

There seems to be just a little that we can glean from the piece - clearly meant for a mass audience. This bit caught my eye:

"He then unveiled his team’s solution: a mathematical model that tries to determine when users want new information and when they don’t. (And yes, like all Google initiatives, it had a name: QDF, for “query deserves freshness.”)
...THE QDF solution revolves around determining whether a topic is “hot.” If news sites or blog posts are actively writing about a topic, the model figures that it is one for which users are more likely to want current information. The model also examines Google’s own stream of billions of search queries, which Mr. Singhal believes is an even better monitor of global enthusiasm about a particular subject."

Matt Cutts of Google added:

To be completely honest, I was a little worried about Saul Hansell, a journalist for the New York Times, sitting in on some of our confidential quality meetings at Google. Even though everything was off-the-record, you can’t help but be slightly nervous talking about evaluation methodologies and confidential projects with a reporter in the room.

- Just because Google doesn’t always talk about search and journalists don’t always write about core search doesn’t mean stuff isn’t happening. Google devotes a ton of effort to improving our search in many different ways.
- Google makes a go/no-go decision on several different quality changes each week.
- If you want to build search loyalty, you have to get a lot of different things right.
- Google has many ways to prioritize feedback and tools to look at how to improve search.
- I’m glad we’re shedding light on some additional people at Google. Many people work behind the scenes to improve the user experience at Google, and we should look to highlight even more of those people.

DigitalPoint Forums has comments as well:

My biggest "wow" came when I read that Google changed their algorithms so frequently (6+ per week).

One item that surprised me was the comment on diversity of results. If after all the rankings are calculated for a search, the top 10 results don't properly reflect the diversity of views on that subject, the 10 results presented will be changed to reflect the diveristy.

Search Engine Watch Forums also has discussion on this article:

Among the topics discussed are how Google handles bug reports, and its own concerns with freshness of its results, including details on the development of the "Query Deserves Freshness (QDF)" solution that determines which queries should show results from new pages, and which should rely on established sites. The article also goes into detail about the 200+ "signals" that are fed into "classifiers" to calculate a page's relevance.

Forum discussion at Search Engine Watch Forums, DigitalPoint Forums and WebmasterWorld.

A Google Groups thread has Google representative Riona MacNamara stating that Google will be Google Webmaster Tools Help.

The most important change they want to make is to improve the way they "organize and present help content," says MacNamara.

So now Google is asking for feedback:

- Do you use the Help Center? Is it easy to find information, and is the information accurate? Is the Help Center -- well, helpful?
- If you don't use it, why not? What other sources of information have been more useful to you? What's missing?
- What's your biggest complaint about the Help Center?
- What do you like best?
- When you go to the Help Center, do you browse the topics? Or do you search?

You can add your two cents at the Google Groups thread.

Forum discussion at Google Groups and Cre8asite Forums.