No abstract

No abstract

Nowadays search engines are recognized as the pathway for accessing the tremendous amount of information in the internet. They provide aids and services for solving users' different information needs. Thus, being able to evaluate their effectiveness and performance is constantly gaining importance because these evaluations are useful for both developers and users of search engines. Developers can use the evaluation results for improving their strategies and paradigms in the development of search engines. Users, on the other hand, can identify the best performing search engines and in a better, quicker and more accurate way, gratify their information needs. Evaluation of search engines can be done in two different ways; either manually using human arbitrators or automatically using automatic machinery approaches which do not use human arbitrators and their judgments. In the case of manual evaluation methods, by now numerous and standard activities had been carried out by organizers and participants of conferences like TREC or CLEF. In the case of automatic evaluation methods, unlike variety of efforts which had been done by different researchers, no categorization and organization of such methods exists so far. As a result, anyone that wants to use one of the automatic evaluation methods must read all the relevant literature of these methods which is very time consuming and confusing activity. In this paper, we have reviewed almost all the important reported automatic methods for evaluation of search engines. Analyzing the results of this review, we have stated the requirements and prerequisites of using any of these methods. At the end, a framework for selecting the best pertinent method for each evaluation scenario has been suggested.Instead, automatic methods of evaluation are cheaper and faster to be performed.Index Terms-Web search engine, information retrieval, automatic evaluation methods.

Measurement of the information retrieval effectiveness of web search engines is an important problem because it can determine which of the available search engines can provide better results to its users. But because of the need of human assessors and their judgments this Measurement is expensive and time consuming. So for solving this problem we introduced an automatic method that is applicable quickly and provides a ranking on the web search engines that were under evaluation without the need of having relevance judgments. In the experiments that were conducted we compared our method with the one proposed by Can et al. and the results acquired from human assessors' evaluations. The comparisons showed that the introduced method provides rankings that are consistent with the rankings resulted from human assessors' evaluations and are better than the ones achieved by the Can et al.'s method. Also it is shown that the consistencies observed are statistically significant, so the proposed method can be used for evaluation of web search engines in real environments.

No abstract