Web crawler (also known as web spider and web robot, and more often called web chaser in FOAF community) is a program or script that automatically crawls information on the World Wide Web according to certain rules. Other less common names are ant, automatic index, emulator or worm.

With the rapid development of the network, the World Wide Web has become the carrier of a large amount of information, and how to effectively extract and use this information has become a huge challenge. Search engines, such as traditional general search engines AltaVista, Yahoo! As a tool to help people retrieve information, Google has become the entrance and guide for users to access the World Wide Web. However, these general search engines also have some limitations, such as:

(1) Users from different fields and backgrounds often have different retrieval purposes and needs, and the results returned by search engines contain a large number of web pages that users don't care about.

(2) The goal of general search engine is to cover as many networks as possible, and the contradiction between limited search engine server resources and unlimited network data resources will be further deepened.

(3) With the rich data forms of the World Wide Web and the continuous development of network technology, a large number of different data such as pictures, databases, audio, video and multimedia appear, and general search engines are often unable to find and obtain these information-intensive and structured data.

(4) Most general search engines provide keyword-based retrieval, and it is difficult to support queries based on semantic information.