文档介绍：: .
摘 要
网络爬虫（Web Crawler），通常被称为爬虫，是搜索引擎的重要组成部分。随着信息技术的飞速进步，作为搜索引擎的一个组成部分——网络爬虫，一直是研究的热点，它的好坏会直接决定搜索引擎的未来。目前，网络爬虫的研究包括Web搜索策略研究的研究和网络分析的算法，两个方向，其中在Web爬虫网络搜索主题是一个研究方向，根据一些网站的分析算法，过滤不相关的链接，连接到合格的网页，并放置在一个队列被抓取。
把互联网比喻成一个蜘蛛网，那么Spider就是在网上爬来爬去的蜘蛛。网络蜘蛛是通过网页的链接地址来寻找网页，从网站某一个页面（通常是首页）开始，读取网页的内容，找到在网页中的其它链接地址，然后通过这些链接地址寻找下一个网页，这样一直循环下去，直到把这个网站所有的网页都抓取完为止。如果把整个互联网当成一个网站，那么网络爬虫就可以用这个原理把互联网上所有的网页都抓取下来。
关键词：网络爬虫；Linux Socket；C/C++;多线程；互斥锁
Abstract
Web Crawler, usually called Crawler for short, is an important part of search engine. With the high-speed development of information, Web Crawler-- the search engine can not lack of-- which is a hot research topic those years. The quality of a search engine is mostly depended on the quality of a Web Crawler. Nowadays, the direction of researching Web Crawler mainly divides into two parts: one is the searching strategy to web pages; the other is the algorithm of analysis URLs. Among them, the research of Topic-Focused Web Crawler is the trend. It uses some webpage analysis strategy to filter topic-less URLs and add fit URLs into URL-WAIT queue.
The metaphor of a spider web internet, then Spider spider is crawling around on the Internet. Web spider through web link address to find pages, starting from a one page website (usually home), read the contents of the page, find the address of the other links on the page, and then look for the next Web page addresses through these links, so has been the cycle continues, until all the pages of this site are crawled exhausted. If the entire Internet as a site, then you can use this Web crawler principle all the pages on the Internet are crawling down..
Keywords:Web crawler；Linux Socket；C/C++; Multithreading;Mutex
目 录
摘 要 0
第一章 概 述 0
课题背景 0
网络爬虫的历史和分类 0
网络爬虫的历史 0
网络爬虫的分类 1
网络爬虫的发展趋势 1
系统开发