Purpose -The purpose of this study is to develop an evaluation criteria system for search engines using the Delphi method and the analytic hierarchy process, and to conduct an empirical study on Chinese search engines to verify it. This is because the evaluation criteria in current research rarely considers expert opinions and experience, and seldom calculates the weight of each criterion by quantitative methods. Design/methodology/approach -Expert investigation of the web-based Delphi method was employed to develop the evaluation criteria, and the analytic hierarchy process based on an exponential scale was adopted to calculate the weight of each criterion. An empirical study was designed to test the evaluation criteria system. A total of 12 study participants were invited to evaluate six popular Chinese search engines by means of e-mail questionnaires. All the calculation processes were executed automatically by Java programming. Findings -The indexing structure and search method were the core criteria for evaluating the search engines. Google (using simplified Chinese characters) achieved the best performance among the six typical Chinese search engines. Originality/value -The criteria for search engine evaluation were determined and weighted by the web-based Delphi method and the analytic hierarchy process based on an exponential scale.
IntroductionWith the rapid development of the internet, web search engines have become more and more important as an information retrieval tool. Seeking information from search engines is now part of people's work and everyday lives. Every day billions of people use search engines to find information they need. The steps involved in using search engines are very similar across the different search engines. The user simply enters a query (typically one or more keywords) into a search engine and then the search engine returns a ranked list of relevant documents or webpages (Lawrence, 2000). However, since the ranking algorithms of the search engines are different (Ogilvie and Callan, 2003), the results lists for a specific query are not the same across multiple search engines. This yields the question of how to evaluate those search engine services.The current modes of evaluating search engines can be divided into two types based on their choice of participants. The first type is third party evaluation, which refers to the evaluation process undertaken by commercial or academic organisations. This kind of evaluation concentrates more on indices such as web traffic, precision rate, etc. The second type is user self-evaluation, which emphasises the selection of appropriate indices based on users' specific information needs. However there are flaws in both these modes. During the construction of the criteria system, third party evaluation usually ignores the users' factors, while the user self-evaluation is often too subjective to construct a stable evaluation criteria system. Moreover the majority of studies evaluating search engines are conducted in an ASCII (e.g. English) enviro...