Eye-Tracking Reveals the Personal Styles for Search Result Evaluation

Aula, Anne; Majaranta, Päivi; Räihä, Kari-Jouko

doi:10.1007/11555261_104

Cited by 90 publications

(85 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Aula et al [3] identified two general patterns that people used in examining search results: exhaustive evaluators (54% of the participants who looked at more than half of the visible results for more than half the tasks) and economic evaluators (46% of the participants). Dumais et al [12] performed a similar analysis of search behavior using more complex result pages that included both organic results and advertisements.…”

Section: Related Workmentioning

confidence: 99%

“…However these studies are typically conducted in laboratories using a small number of participants with assigned tasks (e.g., [9][15]), with summaries of gaze behavior aggregated across participants and tasks. Some studies examined individual and task differences in gaze patterns, and found individual differences in the strategies with which users inspect results [12], and different clusters of users who exhibit similar result examination behaviors [3] [12]. Others found that the type of search task (informational vs. navigational) influenced task completion time and time spent reviewing documents [9] [27].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Large-scale analysis of individual and task differences in search result page examination strategies

Buscher

White

Dumais

et al. 2012

Proceedings of the Fifth ACM International Conference on Web Search and Data Mining

View full text Add to dashboard Cite

Understanding the impact of individual and task differences on search result page examination strategies is important in developing improved search engines. Characterizing these effects using query and click data alone is common but insufficient since they provide an incomplete picture of result examination behavior. Cursor-or gaze-tracking studies reveal richer interaction patterns but are often done in small-scale laboratory settings. In this paper we leverage large-scale rich behavioral log data in a naturalistic setting. We examine queries, clicks, cursor movements, scrolling, and text highlighting for millions of queries on the Bing commercial search engine to better understand the impact of user, task, and user-task interactions on user behavior on search result pages (SERPs). By clustering users based on cursor features, we identify individual, task, and user-task differences in how users examine results which are similar to those observed in small-scale studies. Our findings have implications for developing search support for behaviorally-similar searcher cohorts, modeling search behavior, and designing search systems that leverage implicit feedback.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Large-scale analysis of individual and task differences in search result page examination strategies

Buscher

White

Dumais

et al. 2012

Proceedings of the Fifth ACM International Conference on Web Search and Data Mining

View full text Add to dashboard Cite

show abstract

“…We believe that the user behavior in the Smucker and Jethani study is representative of actual result list processing behavior given the similarity between the study's observed behavior and that reported by other researchers (cf. strategies in Aula [1] and Dumais et al [9], probabilities of clicking on summaries in Yilmaz et al [31]). …”

Section: Calibration Of Model Componentsmentioning

confidence: 99%

Modeling user variance in time-biased gain

Smucker

Clarke

2012

Proceedings of the Symposium on Human-Computer Interaction and Information Retrieval

View full text Add to dashboard Cite

Cranfield-style information retrieval evaluation considers variance in user information needs by evaluating retrieval systems over a set of search topics. For each search topic, traditional metrics model all users searching ranked lists in exactly the same manner and thus have zero variance in their per-topic estimate of effectiveness. Metrics that fail to model user variance overestimate the effect size of differences between retrieval systems. The modeling of user variance is critical to understanding the impact of effectiveness differences on the actual user experience. If the variance of a difference is high, the effect on user experience will be low. Time-biased gain is an evaluation metric that models user interaction with ranked lists that are displayed using document surrogates. In this paper, we extend the stochastic simulation of time-biased gain to model the variation between users. We validate this new version of time-biased gain by showing that it produces distributions of gain that agree well with actual distributions produced by real users. With a per-topic variance in its effectiveness measure, time-biased gain allows for the measurement of the effect size of differences, which allows researchers to understand the extent to which predicted performance improvements matter to real users.

show abstract

“…This is the case, for instance, when studying gaze behaviour on web pages that are composed of horizontally or vertically arranged blocks of information. Another natural application is studies of menu usage [2,3].…”

Section: Static Visualization Of Temporal Gaze Datamentioning

confidence: 99%

“…Time plot visualizations make it possible to visually observe differences in the gaze behaviour of different users. We have used the technique for analyzing how users perceive search result listings (see [2] for more examples).…”

Section: Static Visualization Of Temporal Gaze Datamentioning

confidence: 99%

Static Visualization of Temporal Eye-Tracking Data

Räihä

Aula

Majaranta

et al. 2005

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. Existing static visualization techniques for eye-tracking data do not make it possible to easily compare temporal information, that is, gaze paths. We review existing techniques and then propose a new technique that treats time as the prime attribute to be visualized. We successfully used the new technique for analysing the visual scanning of web search results listings. Based on our experiences, the new visualization is a valuable tool when the temporal order of visiting Areas of Interest (AOI) is the main focus in the analysis, the AOIs have a natural linear order, there are many AOIs to produce interesting patterns, and the AOIs fill most of the coordinate space being studied.

show abstract

Eye-Tracking Reveals the Personal Styles for Search Result Evaluation

Cited by 90 publications

References 7 publications

Large-scale analysis of individual and task differences in search result page examination strategies

Large-scale analysis of individual and task differences in search result page examination strategies

Modeling user variance in time-biased gain

Static Visualization of Temporal Eye-Tracking Data

Contact Info

Product

Resources

About