Indexing

The basis of modern full text search technologies is represented by two fundamental processes. First of all, they are indexing of available information and processing queries with subsequent display of results. As regards indexing, any information retrieval application (be it a desktop full text search system, a corporate informational system or an Internet search engine) creates its own search area. In other words, it processes documents and creates an index of these documents (an organized structure that contains information on processed data). This very index is further used in the work of the search engine for a quick production of a list of required documents according to the query. The rest of the process is not that simple in terms of technology, but is quite simple to understand by a common user. The information retrieval application processes the query (by the key word-phrase) and displays the list of documents that contain this key phrase. Since the information is stored in a structured index, the query is processed much (dozens and hundreds times!) faster than in the case with direct search (the selection of documents is based on the analysis of textual information in the index rather than on browsing through every file).

The full text search application displays the found documents in the resulting list on the relevance basis, i.e., the conformity of the document to the query text. Various search technologies, undoubtedly, comprise various methods of searching and determining the relevance of the document (the number of key word "inclusions" and the frequency of mentioning it in the document, the ratio of these parameters to the general number of words in the document, the distance between the words of the query phrase in the files, and so on). These parameters serve as the basis for determining the "weight" of the document and, depending on this weight, the file ends up in the list of results in a certain position. In case of Internet-search the matter is even more complicated. After all, in this case many other factors have to be taken into account (for example, Page Rank Google). However, this is the subject of a separate article; therefore we will leave the Internet alone for the time being.