Proactive identification of query failure

Liu, Jiqun; Shah, Chirag

doi:10.1002/pra2.15

Cited by 8 publications

(4 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Proactive User Interface and System Design: Our study offers insights for proactive user interface and system design, which extends previous work on proactive IR [15]. Our approach involved utilizing questionnaires and specific interface tabs, yet the need for more seamless integration emerged.…”

Section: Insights and Implications For System Evaluationmentioning

confidence: 71%

Task Supportive and Personalized Human-Large Language Model Interaction: A User Study

Wang,

Liu,

Karimnazarov

et al. 2024

Proceedings of the 2024 ACM SIGIR Conference on Human Information Interaction and Retrieval

Self Cite

View full text Add to dashboard Cite

Large language model (LLM) applications, such as ChatGPT, are a powerful tool for online information-seeking (IS) and problem-solving tasks. However, users still face challenges initializing and refining prompts, and their cognitive barriers and biased perceptions further impede task completion. These issues reflect broader challenges identified within the fields of IS and interactive information retrieval (IIR). To address these, our approach integrates task context and user perceptions into human-ChatGPT interactions through prompt engineering. We developed a ChatGPT-like platform integrated with supportive functions, including perception articulation, prompt suggestion, and conversation explanation. Our findings of a user study demonstrate that the supportive functions help users manage expectations, reduce cognitive loads, better refine prompts, and increase user engagement. This research enhances our comprehension of designing proactive and user-centric systems with LLMs. It offers insights into evaluating human-LLM interactions and emphasizes potential challenges for under served users.CCS Concepts: • Information systems → Retrieval tasks and goals; Personalization; • Human-centered computing → Natural language interfaces.

show abstract

Section: Insights and Implications For System Evaluationmentioning

confidence: 71%

Task Supportive and Personalized Human-Large Language Model Interaction: A User Study

Wang,

Liu,

Karimnazarov

et al. 2024

Proceedings of the 2024 ACM SIGIR Conference on Human Information Interaction and Retrieval

Self Cite

View full text Add to dashboard Cite

show abstract

“…The search path and strategies of users with high performance can be leveraged to enhance other users' search results and experience under similar search tasks, cognitive states, and intents (Hassan Awadallah et al, 2014; Hendahewa & Shah, 2017; J. Liu & Shah, 2022). Users' search performance is shaped by several factors, such as domain knowledge (i.e., users' knowledge background and familiarity with the search task/topic), search expertise (i.e., technical skills and topic‐independent search experience) (Suzuki & Yamamoto, 2021), and contextual factors (J. Liu & Shah, 2019a, 2019b). Focusing on search performance, previous research suggested that good‐performing users usually have better search strategies and can effectively accumulate knowledge and cues through interactive search activities (Mao et al, 2018; Suzuki & Yamamoto, 2021).…”

Section: Introductionmentioning

confidence: 99%

“…Users' search performance is shaped by several factors, such as domain knowledge (i.e., users' knowledge background and familiarity with the search task/topic), search expertise (i.e., technical skills and topic-independent search experience) (Suzuki & Yamamoto, 2021), and contextual factors (J. Liu & Shah, 2019a, 2019b. Focusing on search performance, previous research suggested that good-performing users usually have better search strategies and can effectively accumulate knowledge and cues through interactive search activities (Mao et al, 2018;Suzuki & Yamamoto, 2021).…”

Section: Introductionmentioning

confidence: 99%

Characterizing and Early Predicting User Performance for Adaptive Search Path Recommendation

Wang,

Liu

2023

Proceedings of the Association for Information Science and Tech

Self Cite

View full text Add to dashboard Cite

User search performance is multidimensional in nature and may be better characterized by metrics that depict users' interactions with both relevant and irrelevant results. Despite previous research on one‐dimensional measures, it is still unclear how to characterize different dimensions of user performance and leverage the knowledge in developing proactive recommendations. To address this gap, we propose and empirically test a framework of search performance evaluation and build early performance prediction models to simulate proactive search path recommendations. Experimental results from four datasets of diverse types (1,482 sessions and 5,140 query segments from both controlled lab and natural settings) demonstrate that: 1) Cluster patterns characterized by cost‐gain‐based multifaceted metrics can effectively differentiate high‐performing users from other searchers, which form the empirical basis for proactive recommendations; 2) whole‐session performance can be reliably predicted at early stages of sessions (e.g., first and second queries); 3) recommendations built upon the search paths of system‐identified high‐performing searchers can significantly improve the search performance of struggling users. Experimental results demonstrate the potential of our approach for leveraging collective wisdom from automatically identified high‐performance user groups in developing and evaluating proactive in‐situ search recommendations.

show abstract

“…In addition to advancing science in understanding users, it is also critical to leverage the knowledge about users in providing adaptive and ideally proactive supports for users at early stages of search sessions [3,11,18,22]. [21] found that users' search behaviors in first query of a session can achieve the same level of accuracy in search task prediction as using whole session search data.…”

Section: Introductionmentioning

confidence: 99%

A reference dependence approach to enhancing early prediction of session behavior and satisfaction

Brown

Liu

2022

Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries

Self Cite

View full text Add to dashboard Cite

There is substantial evidence from behavioral economics and decision sciences demonstrating that in the context of decision-making under uncertainty, the carriers of value behind actions are gains and losses defined relative to a reference point (e.g. pre-action expectations), rather than the absolute final outcomes. Also, the capability of early predicting session-level search decisions and user experience is essential for developing reactive and proactive search recommendations. To address these research gaps, our study aims to 1) develop reference dependence features based on a series of simulated user expectations or reference points in first query segments of sessions, and 2) examine the extent to which we can enhance the performance of early predicting session behavior and user satisfaction by constructing and employing reference dependence features. Based on the experimental results on three datasets of varying types, we found that incorporating reference dependent features developed in first query segments into prediction models achieves better performance than using baseline cost-benefit features only in early predicting three key session metrics (user satisfaction score, session clicks, and session dwell time). Also, when running simulations by varying the search time expectation and rate of user satisfaction decay, the results demonstrate that users tended to expect to complete their search within a minute and showed a rapid rate of satisfaction decay in a logarithmic fashion once surpassing the estimated expectation points. By factoring in a user's search time expectation and measuring their behavioral response once the expectation is not met, we can further improve the performance of early prediction models and enhance our understanding of users' behavioral patterns. CCS CONCEPTS• Information systems → Users and interactive retrieval.

show abstract

Proactive identification of query failure

Cited by 8 publications

References 26 publications

Task Supportive and Personalized Human-Large Language Model Interaction: A User Study

Task Supportive and Personalized Human-Large Language Model Interaction: A User Study

Characterizing and Early Predicting User Performance for Adaptive Search Path Recommendation

A reference dependence approach to enhancing early prediction of session behavior and satisfaction

Contact Info

Product

Resources

About