Interactive IR User Study Design, Evaluation, and Reporting

Liu, Jiqun; Shah, Chirag

doi:10.2200/s00923ed1v01y201905icr067

Cited by 15 publications

(16 citation statements)

References 111 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Search results exploration is the most challenging phase in the information-seeking process [55]. Adding to that, listening to a screen-reader during the exploration phase increases the user's cognitive load, and cognitive load is a fundamental aspect of search system design [65]. In this section, we compare the Google search engine interface and InteractSE interface when explored using a screen reader in Figs.…”

Section: Search Results Presentation and Interactive Explorationmentioning

confidence: 99%

Can search result summaries enhance the web search efficiency and experiences of the visually impaired users?

Aqle

Al-Thani

Jaoua

2020

Univ Access Inf Soc

View full text Add to dashboard Cite

There are limited studies that are addressing the challenges of visually impaired (VI) users when viewing search results on a search engine interface by using a screen reader. This study investigates the effect of providing an overview of search results to VI users. We present a novel interactive search engine interface called InteractSE to support VI users during the results exploration stage in order to improve their interactive experience and web search efficiency. An overview of the search results is generated using an unsupervised machine learning approach to present the discovered concepts via a formal concept analysis that is domain-independent. These concepts are arranged in a multi-level tree following a hierarchical order and covering all retrieved documents that share maximal features. The InteractSE interface was evaluated by 16 legally blind users and compared with the Google search engine interface for complex search tasks. The evaluation results were obtained based on both quantitative (as task completion time) and qualitative (as participants’ feedback) measures. These results are promising and indicate that InteractSE enhances the search efficiency and consequently advances user experience. Our observations and analysis of the user interactions and feedback yielded design suggestions to support VI users when exploring and interacting with search results.

show abstract

Section: Search Results Presentation and Interactive Explorationmentioning

confidence: 99%

Can search result summaries enhance the web search efficiency and experiences of the visually impaired users?

Aqle

Al-Thani

Jaoua

2020

Univ Access Inf Soc

View full text Add to dashboard Cite

show abstract

“…Perhaps most closely aligned is a pocket of literature in information science relating to user-centered interactive information retrieval (IIR) which takes a cognitive viewpoint and shifts study focus to the user’s role as opposed to the system (Belkin, 1990). These studies often centralize around understanding user behavior and experience, evaluating IR system or interface features, and meta-evaluation of evaluation metrics (Liu and Shah, 2019). For a comprehensive overview of IIR current study designs and methods, readers should refer to Kelly (2009), Kelly and Sugimoto (2013), and Liu and Shah (2019).…”

Section: Introductionmentioning

confidence: 99%

“…These studies often centralize around understanding user behavior and experience, evaluating IR system or interface features, and meta-evaluation of evaluation metrics (Liu and Shah, 2019). For a comprehensive overview of IIR current study designs and methods, readers should refer to Kelly (2009), Kelly and Sugimoto (2013), and Liu and Shah (2019). User-centered IIR often employs various methodological components that align with Gist et al’s (1998) definition of behavioral simulation.…”

Section: Introductionmentioning

confidence: 99%

How real is real enough? Participant feedback on a behavioral simulation used for information-seeking behavior research

Buhler

Brannon

Cataldo

et al. 2022

Journal of Librarianship and Information Science

View full text Add to dashboard Cite

A challenge of studying information-seeking behavior in open web systems is the unpredictability of those systems. One solution to counteract this issue is employing a simulation to ensure experimental control. However, concerns arise over the realism of such an environment. This paper assesses the realism of a behavioral simulation used to study the evaluation behavior of 175 students from fourth grade through graduate school. We assess realism through the examination of targeted participant feedback about what would have made the simulated environment and tasks more realistic to these participants. Based on this feedback, we reflect on decisions made in designing the simulation and offer recommendations for future studies interested in incorporating behavioral simulation in their research design. We find that a thoughtfully designed simulation can elicit naturalistic behavior when the controlled environment is designed to be realistic in meaningful ways. Because the simulation does not have to perfectly match reality to elicit these behaviors, designing a simulation that is real enough is an effective method to study information-seeking behavior.

show abstract

“…For that reason, as an approach focusing on users and user-system interactions, user study is a critical part of IIR research. Indeed, it’s the only way to discover exactly what these users expect, think and need, addressing a variety of research questions that are directly or indirectly related to search system design and evaluation (Liu and Shah, 2019) and bridge the gap between the kind of information services needed and the kind of services in existence (Satyanarayana, 2017). Moreover, it can remove bias by learning about the users from their experiences or perspectives and help to ground design decisions based on a good grip on users’ needs (Yang, 2019).…”

Section: Introductionmentioning

confidence: 99%

Reflection on future directions: a systematic review of reported limitations and solutions in interactive information retrieval user studies

Jiang

Liu

2022

AJIM

View full text Add to dashboard Cite

PurposeThe purpose of this paper is to understand how users behave and evaluate how systems with users are essential for interactive information retrieval (IIR) research. User study methodology serves as a primary approach to answering IIR research questions. In addition to designing user study procedures, understanding the limitations of varying study designs and discussing solutions to the limitations is also critical for improving the methods and advancing the knowledge in IIR.Design/methodology/approachGiven this unresolved gap, we apply the faceted framework developed by Liu and Shah (2019) in systematically reviewing 131 IIR user studies recently published (2016–2021) on multiple IR and information science venues.FindingsOur study achieve three goals: (1) extracting and synthesizing the reported limitations on multiple aspects of user study (e.g. recruitment, tasks, study procedures, system interfaces, data analysis methods) under associated facets; (2) summarizing the reported solutions to the limitations; (3) clarifying the connections between types of limitations and types of solutions.Practical implicationsThe bibliography of user studies can be used by students and junior researchers who are new to user-centered IR studies as references for study design. Our results can facilitate the reflection and improvement on IR research methodology and serve as a checklist for evaluating customized IIR user studies in varying problem spaces.Originality/valueTo our knowledge, this work is the first study that systematically reviews the study limitations and solutions reported by IIR researchers in articles and empirically examines their connections to different study components.

show abstract

Interactive IR User Study Design, Evaluation, and Reporting

Cited by 15 publications

References 111 publications

Can search result summaries enhance the web search efficiency and experiences of the visually impaired users?

Can search result summaries enhance the web search efficiency and experiences of the visually impaired users?

How real is real enough? Participant feedback on a behavioral simulation used for information-seeking behavior research

Reflection on future directions: a systematic review of reported limitations and solutions in interactive information retrieval user studies

Contact Info

Product

Resources

About