A previous article about Chinese word segmentation technology in php.
abstract:
In this paper, based on the technology of Chinese full-text search on the Website, combined with the requirements of PHP(PHP:Hypertext Preprocessor) on the performance and memory consumption of web applications in practical fields, a lightweight and efficient solution of Chinese search engine on the website based on pure PHP pre-index dictionary is proposed.
Main contents: The indexer saves the weighted index and word frequency weighted index of the generated full-text data in the database. Based on these full-text data, the retriever can calculate the relevance according to the weight definitions of multiple categories to get search results, and the indicator will highlight and sort the results and return them to the search users to complete the search function.
As the core of Chinese data processing, Chinese word segmenter based on massive dictionaries correctly segments Chinese, English and digital information, and enables indexer to index according to lexical weight, thus realizing rich and flexible search or indexing related functions.
This paper studies the three most prominent aspects of Chinese search technology in PHP station.
1) Lightweight and efficient design of PHP Chinese search framework, and unified consideration of indexer and Chinese word segmentation of indexer, so that the same word segmentation results can be processed during indexing and searching. In this way, the accuracy of word segmentation is guaranteed to be above 90% at a very small cost, and at the same time, it has a good tolerance for inaccurate word segmentation results, ensuring the lightweight and ease of use of PHP applications. It has certain reference significance for the design and development of Web applications that are very sensitive to performance in practice.
2) A method of calculating the correlation of data search results in the station with multiple weight factors is proposed. On the basis of traditional keyword weight relevance, this method combines HTML tags to identify and count the weights, and increases the relevant weight factors that users can intervene through categories such as document attributes and statistical data, which effectively ensures the effectiveness of search results and improves the search experience of users in the station.
3) In order to improve the quality of Chinese word segmentation and solve the problems of performance and memory consumption when dealing with a large number of dictionaries in PHP application, this paper uses an optimized word segmentation matching algorithm and a B-tree pre-index dictionary innovatively, using more than 530,000 UTF-8 simplified and traditional Chinese words, which ensures a good word segmentation result while keeping Chinese search light and efficient. Practice has proved that the algorithm has good usability and universality, and has low algorithm time complexity.
Innovation:
Based on PHP technology, search engine and Chinese word segmentation, this paper puts forward an effective analysis and solution to realize lightweight and efficient Chinese search in PHP field.
With the continuous development of Web applications, the extensive application of PHP and the increasing demand for Chinese information processing, the methods discussed in this paper have certain guiding significance for Chinese search or indexing related functions within the scope of PHP.
At the same time, with the evolution of advanced search engine technology mode, the analysis and research done in this paper have made a meaningful exploration for the universal application of Chinese website search.
Please refer to it.