文档介绍：近似镜像网页检测算法及其评测
摘要
随着Web上网页数量的急剧增长,近似镜像(near-replicas)网页的数量也在不断增加。近似镜像网页的存在,严重影响了搜索引擎的检索结果。如果我们能将搜集到的网页中的近似镜像网页去掉,可以提高搜集系统和索引系统效率,用户查询时也不会出现大量内容重复的网页。去除镜像网页的算法即是近似镜像网页检测算法。本文在大量实验和真实数据分析的基础上,得到了如下研究成果和结论:
提出了一种全文分块签名的多指纹近似镜像网页检测算法。该算法利用网页的标签树结构选取文本块生成MD5指纹序列,先通过部分MD5的预消重处理,然后将预消重处理的结果网页对两两比较得到算法的最终结果。设网页个数为N,每个网页的MD5序列长度为m,则算法的时间和空间复杂度分别是和。
设定了针对近似镜像网页检测算法的评测与比较方法。该方法延续了传统的近似镜像网页检测算法的评测方法,并且重点分析了不同近似镜像检测算法结果的差集,提出了相对精确比ρ的概念,并以次作为算法优劣性的评判标准。
本文在1,000,000网页的数据集内评测了全文分块签名的多指纹近似将向网页检测算法的性能,并利用天网现用的近似镜像网页检测算法作为基准程序,进行了优劣性的对比分析。
关键词:万维网、搜索引擎、近似镜像、MD5指纹
Abstract
There are numerous near-replicas web pages on the World Wide Web, how to efficiently and accurately find the near-replicas web pages has e an important topic in the search engine research area. If we can detect the near-replicas and remove it from the database, we could improve the quality of crawling system and searching service, and the users would be more satisfied. Based on a great deal of experiments and analysis of real data, this article received the following achievements and conclusion.
1) proposes a whole-text partition near-replicas detecting algorithm which uses multiply MD5 fingerprints. The algorithm first gains MD5s from the tag tree of a HTML web page, then pre-detecting replications from the first some MD5s, and then get the final result by pare all MD5s of web pages in the pre-detecting result. Assume the dataset has N web pages, and each web page has m MD5 fingerprints, the time and plexity of the algorithm is
2) formulates the evaluating system for near-replicas detecting algorithms. By analyzing the difference set of two algorithm's result, the author put forward a new measure of relative precision rate pare and contrast such algorithms. The system is first used in this paper to evaluating the near-replicas detecting algorithms.
3) evaluates and analyzes our algorithm according to the evaluating system on the data set of 1,000,000 web pages, using the WebGather search engine base sys