Using a Delphi method and the analytic hierarchy process to evaluate Chinese search engines

Abstract

Purpose -The purpose of this study is to develop an evaluation criteria system for search engines using the Delphi method and the analytic hierarchy process, and to conduct an empirical study on Chinese search engines to verify it. This is because the evaluation criteria in current research rarely considers expert opinions and experience, and seldom calculates the weight of each criterion by quantitative methods. Design/methodology/approach -Expert investigation of the web-based Delphi method was employed to develop the evaluation criteria, and the analytic hierarchy process based on an exponential scale was adopted to calculate the weight of each criterion. An empirical study was designed to test the evaluation criteria system. A total of 12 study participants were invited to evaluate six popular Chinese search engines by means of e-mail questionnaires. All the calculation processes were executed automatically by Java programming. Findings -The indexing structure and search method were the core criteria for evaluating the search engines. Google (using simplified Chinese characters) achieved the best performance among the six typical Chinese search engines. Originality/value -The criteria for search engine evaluation were determined and weighted by the web-based Delphi method and the analytic hierarchy process based on an exponential scale. IntroductionWith the rapid development of the internet, web search engines have become more and more important as an information retrieval tool. Seeking information from search engines is now part of people's work and everyday lives. Every day billions of people use search engines to find information they need. The steps involved in using search engines are very similar across the different search engines. The user simply enters a query (typically one or more keywords) into a search engine and then the search engine returns a ranked list of relevant documents or webpages (Lawrence, 2000). However, since the ranking algorithms of the search engines are different (Ogilvie and Callan, 2003), the results lists for a specific query are not the same across multiple search engines. This yields the question of how to evaluate those search engine services.The current modes of evaluating search engines can be divided into two types based on their choice of participants. The first type is third party evaluation, which refers to the evaluation process undertaken by commercial or academic organisations. This kind of evaluation concentrates more on indices such as web traffic, precision rate, etc. The second type is user self-evaluation, which emphasises the selection of appropriate indices based on users' specific information needs. However there are flaws in both these modes. During the construction of the criteria system, third party evaluation usually ignores the users' factors, while the user self-evaluation is often too subjective to construct a stable evaluation criteria system. Moreover the majority of studies evaluating search engines are conducted in an ASCII (e.g. English) enviro...

Inter-Campus Sharable Potential of Hardware Educational Resources in A University Town: Connotation, Determination Method and A Case Study

“…The index weight (Wi) is determined by the Delphi method [49], and the weights of the evaluation buildings are between 0 and 1, and their sum is 1. We calculate the weight value according to Equation (6).…”

“…The weights of HERs are determinated by the Delphi method [49]. After three rounds of scoring by 20 experts, the weights calculated by Equation (6) are 0.3135 for classrooms, 0.2039 for laboratories, 0.1994 for libraries, 0.2832 for canteens and 0.198 for indoor sports rooms, respectively.…”

Penentuan Faktor Pendorong Penyebaran Wabah Penyakit Berdasarkan Metode Ahp - Delphi (Studi Kasus: KLB Leptospirosis Di Kabupaten Bantul)

“…Setelah menguji konsistensi, semua penilaian diagregasikan dengan ratarata geometris yang merupakan aturan yang tepat untuk menggabungkan penilaian individu untuk mendapatkan penilaian kelompok untuk setiap perbandingan yang bijaksana (Byun, 2001). Jika masing-masing pembuat keputusan memiliki inkonsistensi yang dapat diterima, hasil dalam keputusan kelompok juga dapat diterima (Escobar et al, 2004;Taleai and Mansourian, 2008 (Hsu and Sandford, 2007;Zhu et al, 2011).…”

A risk appraisal system regarding the implementation of maritime regulations by a ship operator

“…The Delphi technique is a structured process which allows experts to deal systematically with complex tasks, by means of controlled feedback and statistical response, and is recognized as an appropriate research tool where exploration of ideas and production of suitable information for decision making are required, and its adoption is particularly indicated in case of complex, interdisciplinary problems involving several new concepts (Bigliardi 2012). The Delphi method is not only used in forecasting, but also widely adopted in criteria system construction (Zhu et al 2011). The Delphi method consists of many rounds of surveys until experts reach an agreement for their judgments.…”