文档介绍：:通用爬虫、聚焦爬虫、,所以搜索操作可以总结为对有向图的遍历。爬虫通过web页面的图结构从一个页面到另一个页面。,其来源于google的pagerank算法和锚点词权重。:A,通用爬虫:从每一个网页尽可能多的找寻链接,去点速度慢占带宽。B,聚焦爬虫:爬取某一主题的文件,节省带宽。C分布式爬虫:多线程爬取。,有重载、质量和网络等问题。,,’00,,J.,,,:repeatqueriesinYahoo'‘07,151-158,。,,,,“DesignandImplementationofWebCrawlerBasedonDynamicWebCollectionCycle”,-566,IEEE2012动态web数据爬取技术包含对web变化的监视,动态获取网页。arcia-Molina“ParallelCrawlers”.Proceedingsofthe11thinternationalconferenceonWorldWideWebWWW'02”,May7–11,2002,Honolulu,Hawaii,-58113-449-5/02/,RaviKumarP,AshutoshKumarSinghandRajendraKumarDash“PyBot:AnAlgorithmforWebCrawling”,IEEE2011广度优先搜素,会输出一个ExcelCSV形式的web架构,存储下来的网页与web结构用于排名,RajashreeShettar,,“WebCrawlerOnClientMachine”,puterScientists2008VolIIIMECS2008,19-21March,2008,HongKong异步多线程下载模块EytanAdar,JaimeTeevan,“TheWebChangesEverything:UnderstandingtheDynamicsofWebContent”,。,“PARCAHYD:AnArchitectureofaParallelCrawlerbasedonAugmentedHypertextDocuments”,InternationalJournalofAdvancementsinTechnology,-283,,分