The Effect of Document Order and Topic Difficulty on Assessor Agreement

Damessie, Tadele T.; Scholer, Falk; Järvelin, Kalvero; Culpepper, J. Shane

doi:10.1145/2970398.2970431

Cited by 7 publications

(2 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many researchers in information science, such as Resnick (1961), Saracevic (1995), Tague-Sutcliffe (1996), Chu and Rosenthal (1996), Hawking et al (1999), Oppenheim et al (2000), Voorhees (2001), MacFarlane (2007), Bilal (2012), Lewandowski (2015), and Damessie et al (2016), discuss about “evaluation.” Evaluation means assessing the performance or value of a system, process (technique, procedure…), product, or policy (Saracevic, 1995). The value of the information retrieval system, especially the search engines, is to retrieve documents relevant to users' needs (Cooper, 1971; Voorhees, 2001; Saracevic, 2007).…”

Section: Introductionmentioning

confidence: 99%

“…Voorhees (2001) discussed the fundamental assumptions and appropriate uses of the Cranfield paradigm. Damessie et al (2016) examined the effect of document order and topic difficulty on judges. Saracevic has written several articles (1995, 2007, 2012; 2015) that have accounted for a critical and historical analyses of evaluations of IR systems and processes.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Evaluating the effectiveness of Google, Parsijoo, Rismoon, and Yooz to retrieve Persian documents

Tazehkandi

Nowkarizi

2020

LHT

View full text Add to dashboard Cite

PurposeThe purpose was to evaluate the effectiveness of Google (as an international search engine) as well as of Parsijoo, Rismoon, and Yooz (as Persian search engines).Design/methodology/approachIn this research, Google search engine as an international search engine, and three local ones, Parsijoo, Rismoon, and Yooz, were selected for evaluation. Likewise, 32 subject headings were selected from the Persian Subject Headings List, and then simulated work tasks were assigned based on them. A total of 192 students from Ferdowsi University of Mashhad were asked to search for the information needed for simulated work tasks in the selected search engines, and then to copy the relevant website URLs in the search form.FindingsThe findings indicated that Google, Parsijoo, Rismoon, and Yooz had a significant difference in the precision, recall, and normalized discounted cumulative gain. There was also a significant difference in the effectiveness (average of precision, recall, and NDCG) of these four search engines in the retrieval of the Persian resources.Practical implicationsUsers using an efficient search engine will attain more relevant documents, and Google search engine was more efficient in retrieving the Persian resources. It is recommended to use Google as it has a more efficient search.Originality/valueIn this research, for the first time, Google has been compared with local Persian search engines considering the new approach (simulated work tasks).

show abstract