Crowdsourcing interactions: using crowdsourcing for evaluating interactive information retrieval systems

Zuccon, Guido; Leelanupab, Teerapong; Whiting, Stewart; Yılmaz, Emine; Jose, Joemon M.; Azzopardi, Leif

doi:10.1007/s10791-012-9206-z

Cited by 48 publications

(37 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As highlighted by Zuccon et al [54], crowdsourcing provides an alternative means for capturing user interactions and search behaviours from traditional lab-based user studies. Greater volumes of data can be obtained from more heterogeneous workers at a lower cost -all within a shorter timeframe.…”

Section: Crowdsourced Subjects and Ality Controlmentioning

confidence: 99%

“…Of course, pitfalls of a crowdsourced approach include the possibility of workers completing tasks as e ciently as possible, or submi ing their tasks without performing the requested operations [13]. Despite these issues, it has been shown that there is li le di erence in the quality between crowdsourced and lab-based studies [54]. Nevertheless, quality control is a major component of a well-executed crowdsourced experiment [5].…”

Section: Crowdsourced Subjects and Ality Controlmentioning

confidence: 99%

See 1 more Smart Citation

A Study of Snippet Length and Informativeness

Maxwell

Azzopardi

Moshfeghi

2017

Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

Self Cite

View full text Add to dashboard Cite

e design and presentation of a Search Engine Results Page (SERP) has been subject to much research. With many contemporary aspects of the SERP now under scrutiny, work still remains in investigating more traditional SERP components, such as the result summary. Prior studies have examined a variety of di erent aspects of result summaries, but in this paper we investigate the in uence of result summary length on search behaviour, performance and user experience. To this end, we designed and conducted a withinsubjects experiment using the TREC AQUAINT news collection with 53 participants. Using Kullback-Leibler distance as a measure of information gain, we examined result summaries of di erent lengths and selected four conditions where the change in information gain was the greatest: (i) title only; (ii) title plus one snippet; (iii) title plus two snippets; and (iv) title plus four snippets. Findings show that participants broadly preferred longer result summaries, as they were perceived to be more informative. However, their performance in terms of correctly identifying relevant documents was similar across all four conditions. Furthermore, while the participants felt that longer summaries were more informative, empirical observations suggest otherwise; while participants were more likely to click on relevant items given longer summaries, they also were more likely to click on non-relevant items. is shows that longer is not necessarily be er, though participants perceived that to be the case -and second, they reveal a positive relationship between the length and informativeness of summaries and their a ractiveness (i.e. clickthrough rates). ese ndings show that there are tensions between perception and performance when designing result summaries that need to be taken into account.

show abstract

Section: Crowdsourced Subjects and Ality Controlmentioning

confidence: 99%

Section: Crowdsourced Subjects and Ality Controlmentioning

confidence: 99%

A Study of Snippet Length and Informativeness

Maxwell

Azzopardi

Moshfeghi

2017

Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

Self Cite

View full text Add to dashboard Cite

show abstract

“…Nevertheless, due to the recent nature of this type of citizen participation initiatives, there is little empirical research about its effectiveness. In a recent study, Zuccon et al [78] have compared the effectiveness of crowdsourcing versus traditional methods (laboratory-based user studies) in the evaluation of interactive information retrieval systems. Their results show that crowdsourcing provides an opportunity of reaching a higher number of participants and provides results as valuable as those provided by traditional methods, at half the cost and collecting five times more data.…”

Section: Crowdsourcing: Models and Impact Assessmentmentioning

confidence: 99%

“…From a practical perspective, this means that managers have to decide whether they are more interested in the quantity of ideas or in the quality of the ideas. Zuccon et al [78] note that a more effective strategy is to combine both methodologies: traditional methods for pilot studies followed by crowdsourcing, which allows access to a larger number of potential participants.…”

Section: Crowdsourcing: Models and Impact Assessmentmentioning

confidence: 99%

“Crowdsourcing” as a tool for e-participation: two experiences regarding CO2 emissions at municipal level

Royo

Yetano

2015

Electron Commer Res

View full text Add to dashboard Cite

Public administrations need to find innovative approaches to solving changing problems and to fulfilling their duties. Crowdsourcing in local governments could be defined as the use of new technologies in order to take advantage of citizen knowledge to find solutions to municipal problems. This paper analyzes two experiences of crowdsourcing at local level, and the opinions of the citizens and managers involved, to provide an initial assessment of its effectiveness and suggest avenues for improvement. Engaging citizens in e-participation initiatives is a challenge but, for public sector crowdsourcing initiatives, intrinsic motivation and intangible rewards seem to be enough to motivate participants. Results show that managers find citizens' propositions viable, but publishing the final results and indicating how the ideas selected will be further developed are pending tasks. Caution is needed as failure to deliver the benefits deemed essential by citizens can make them even more reluctant to engage.

show abstract

“…A variety of recent work has investigated crowdsourcing methods for IR data collection [1], both for Cranfield-style relevance judging [35] and interactive IR studies [78]. Potential benefits include faster, easier, cheaper, and scalable data collection, increasing diversity of data available beyond that provided by traditional assessors and university students, and the potential greater similarity between the crowd and typical IR system users.…”

Section: Crowdsourcingmentioning

confidence: 99%

Multidimensional relevance modeling via psychometrics and crowdsourcing

Zhang

Lease

et al. 2014

Proceedings of the 37th International ACM SIGIR Conference on Research &Amp; Development in Information Retrieval

View full text Add to dashboard Cite

While many multidimensional models of relevance have been posited, prior studies have been largely exploratory rather than confirmatory. Lacking a methodological framework to quantify the relationships among factors or measure model fit to observed data, many past models could not be empirically tested or falsified. To enable more positivist experimentation, Xu and Chen [77] proposed a psychometric framework for multidimensional relevance modeling. However, we show their framework exhibits several methodological limitations which could call into question the validity of findings drawn from it. In this work, we identify and address these limitations, scale their methodology via crowdsourcing, and describe quality control methods from psychometrics which stand to benefit crowdsourcing IR studies in general. Methodology we describe for relevance judging is expected to benefit both human-centered and systems-centered IR.

show abstract

Crowdsourcing interactions: using crowdsourcing for evaluating interactive information retrieval systems

Cited by 48 publications

References 8 publications

A Study of Snippet Length and Informativeness

A Study of Snippet Length and Informativeness

“Crowdsourcing” as a tool for e-participation: two experiences regarding CO2 emissions at municipal level

Multidimensional relevance modeling via psychometrics and crowdsourcing

Contact Info

Product

Resources

About