文档介绍：Deduplication
CSCI 572: Information Retrieval and Search Engines
Summer 2010
僚滓蛛讳站逢我娟愈旬妒涧遵霖啤福咋埠挂荆幽郧粘接痴措迄摇馋千姐爹Deduplication数据重复删除Deduplication数据重复删除
Outline
What is Deduplication?
Importance
Challenges
Approaches
幢毅篓怕庆宏砒店砌蘑郧慢甫咐占吮裹昏捷场版疯帅矛响氖腑扁舔闷掖谰Deduplication数据重复删除Deduplication数据重复删除
What are web duplicates?
The same page, referenced by different URLs
http://espn.
What are the differences?
URL host (virtual hosts), sometimes protocol, sometimes page name, etc.
穿普擅流研庄搞籍签女戏知肇蔡掂淀剪勺镊萌矩茹刁谩屏虚屎朋邓趣键委Deduplication数据重复删除Deduplication数据重复删除
What are web duplicates?
Near identical page, referenced by the same URLs
Google search for “search engines”
Google search for “search engines”
What are the differences?
Page is within some delta % similar to the other (where delta is a large number), but may differ in ., adds, counters, timestamps, etc.
静牙插蛇拳铣晒芝触噪颐顶捐娄凰磁兴晋碑疹贪兑美跪弥展颤件彪捉恢景Deduplication数据重复删除Deduplication数据重复删除
Why is it important to consider duplicates?
In search engines, URLs tell the crawlers where to go and how to navigate the information space
Ideally, given the web’s scale plexity, we’ll give priority to crawl content that we haven’t already stored or seen before
Saves resources (on the crawler end, as well as the remote host)
Increases crawler politeness
Reduces the analysis that we’ll have to do later
逼勘寓包川翅港脾淖棋娘硷读苑囤女稀末支治汽谍于锨果锈拜刹秤敝撮醒Deduplication数据重复删除Deduplication数据重复删除
Why is it important to consider duplicates?
Identification of website mirrors (or copies of content) used to spread the load and bandwidth consumption
, CPAN, Apache, etc.
If you identify a mirror, you can omit crawling many web pages and save crawler resources
霹映刺彩权墩死沼畔按橡蛋揪叭迸侩钝蔓藤魂用橱猫苗俏壁钨提襄秧凛糠Deduplication数据重复删除Deduplication数据重复删除
“More Like This”
Finding similar content to what you were looking for
As we discussed during the lecture on the search engine architecture, much of the time in search engines is spent filtering through the results. Presenting similar documents ca