SpiderDuck: The Realtime Twitter Spider

Nov 18, 2011 - 8:30 am 3 by
Filed Under Spiders

Twitter SpiderDuckTwitter has unveiled their URL fetcher, which they named SpiderDuck.

What does it do?

SpiderDuck is a service at Twitter that fetches all URLs shared in Tweets in real-time, parses the downloaded content to extract metadata of interest and makes that metadata available for other Twitter services to consume within seconds.

Twitter says this helps them improve the following Twitter services:

  • Search to index resolved URLs and improve relevance
  • Clients to display certain types of media, such as photos, next to the Tweet
  • Tweet Button to count how many times each URL has been shared on Twitter
  • Trust & Safety to aid in detecting malware and spam
  • Analytics to surface a variety of aggregated statistics about links shared on Twitter

A WebmasterWorld thread has some webmasters sharing the useragents and such found in their log files.

Anyway, for more details see the Twitter post.

Forum discussion at WebmasterWorld.

Image credit: Yayayoyo & YorkBerlin / ShutterStock.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google Core Update Volatility, Helpful Content Update Gone, Dangerous Search Results & Ads Confusion - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: April 12, 2024

Apr 12, 2024 - 4:00 pm
Search Video Recaps

Search News Buzz Video Recap: Google Core Update Volatility, Helpful Content Update Gone, Dangerous Google Search Results & Google Ads Confusion

Apr 12, 2024 - 8:01 am
Google Search Engine Optimization

Google: Indexing & Algorithm Updates Are Independent

Apr 12, 2024 - 7:51 am
Google Search Engine Optimization

Google Structured Data Carousels Beta Docs Clarifies Feature Availability & Markup Location

Apr 12, 2024 - 7:41 am

Google: Ranking In Shopping, Images & Other Verticals Doesn't Hurt Your Web Rankings

Apr 12, 2024 - 7:31 am

Google Knowledge Panels - Mentioned People

Apr 12, 2024 - 7:21 am
Previous Story: Louis Daguerre Google Logo