Deep Sequential Models for Task Satisfaction Prediction

Mehrotra, Rishabh; Awadallah, Ahmed Hassan; Shokouhi, Milad; Yılmaz, Emine; Zitouni, Imed; Kholy, Ahmed El; Khabsa, Madian

doi:10.1145/3132847.3133001

Cited by 22 publications

(5 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There is limited research aiming at understanding the consequences of user abandoning HITs in crowdsourcing marketplaces. Some existing studies on satisfaction have tried to analyze user interaction from different dimensions to improve their search experience, e.g., [22], [24], [31], [32]. Differently to them, we focus on crowdsourcing workers who give up before completing their HITs aiming at understanding task abandonment on crowdsourcing platforms by examining their interaction and behavior while working on tasks.…”

Section: Related Workmentioning

confidence: 99%

The Impact of Task Abandonment in Crowdsourcing

Han

Roitero

Gadiraju

et al. 2019

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

Crowdsourcing has become a standard methodology to collect manually annotated data such as relevance judgments at scale. On crowdsourcing platforms like Amazon MTurk or FigureEight, crowd workers select tasks to work on based on different dimensions such as task reward and requester reputation. Requesters then receive the judgments of workers who self-selected into the tasks and completed them successfully. Several crowd workers, however, preview tasks, begin working on them, reaching varying stages of task completion without finally submitting their work. Such behavior results in unrewarded effort which remains invisible to requesters. In this paper, we conduct an investigation of the phenomenon of task abandonment, the act of workers previewing or beginning a task and deciding not to complete it. We follow a three-fold methodology which includes 1) investigating the prevalence and causes of task abandonment by means of a survey over different crowdsourcing platforms, 2) data-driven analysis of logs collected during a large-scale relevance judgment experiment, and 3) controlled experiments measuring the effect of different dimensions on abandonment. Our results show that task abandonment is a widely spread phenomenon. Apart from accounting for a considerable amount of wasted human effort, this bears important implications on the hourly wages of workers as they are not rewarded for tasks that they do not complete. We also show how task abandonment may have strong implications on the use of collected data (for example, on the evaluation of Information Retrieval systems).

show abstract

Section: Related Workmentioning

confidence: 99%

The Impact of Task Abandonment in Crowdsourcing

Han

Roitero

Gadiraju

et al. 2019

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

show abstract

“…Jiang et al (Jiang et al, 2015) argued that satisfaction could be expressed in different levels and proposed a model that predicted searcher satisfaction at multiple levels. Lastly, work from Mehrotra et al (2017) applied deep sequential models and novel machine learning techniques to predict satisfaction. Long Papers…”

Section: Related Work Satisfaction and Irmentioning

confidence: 99%

Neural Correlates of Realisation of Satisfaction in a Successful Search Process

Paisalnan

Moshfeghi

Pollick

2021

Proceedings of the Association for Information Science and Tech

View full text Add to dashboard Cite

In a search process, searchers review documents to gather information relevant to their information need (IN). During this process, searchers may experience the satisfaction of their IN, indicating they have gathered adequate relevant information to answer their need. This complex concept of satisfaction is the ultimate goal of search systems. Most studies in Information Retrieval (IR) have been attempted to understand how searchers' needs are satisfied based on behavioural observation. However, the psychophysiological manifestation during the moment of satisfaction still remains unclear. Here, we use functional Magnetic Resonance (fMRI) to investigate which brain regions are involved during the moment of satisfaction. Twenty-six participants participated in the experiment, designed to represent a search process while being scanned. Our result shows the human brain regions involved during the moment of satisfaction. These findings provide an important step in unravelling the concept of satisfaction in a search process.

show abstract

“…In traditional information retrieval literature, there are extensive existing work in predicting user satisfaction with a searching query under different user intents [3,17,26,27]. In [26], the authors considered the keywords submitted in an ad auction as a direct expression of intent and use them to determine the relevance of the ad and user query.…”

Section: User Intent Modeling and Satisfaction Predictionmentioning

confidence: 99%

“…Although advertising expenses constitute a major revenue income of e-commerce platforms, little attention has been given to investigate advertiser's behavior, intent or satisfaction either from academia or industry community. Previous studies mainly focused on modeling and understanding the behaviors of users, such as widely-studied Click-Through Rate (CTR) prediction [4,6,9,11,30,31] and user intent or satisfaction estimation in diverse contexts [3,12,17,18,25,27]. As an ad platform can be modeled as a two-sided market between users and advertisers [8], ignoring the actions and feedback from advertisers would significantly reduce the efficiency of user-advertiser matching, and deteriorate the performance of ad systems in the long term.…”

Section: Introductionmentioning

confidence: 99%

A Deep Prediction Network for Understanding Advertiser Intent and Satisfaction

Guo

Lü

Zhang

et al. 2020

Proceedings of the 29th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

For e-commerce platforms such as Taobao and Amazon, advertisers play an important role in the entire digital ecosystem: their behaviors explicitly influence users' browsing and shopping experience; more importantly, advertiser's expenditure on advertising constitutes a primary source of platform revenue. Therefore, providing better services for advertisers is essential for the long-term prosperity for e-commerce platforms. To achieve this goal, the ad platform needs to have an in-depth understanding of advertisers in terms of both their marketing intents and satisfaction over the advertising performance, based on which further optimization could be carried out to service the advertisers in the correct direction. In this paper, we propose a novel Deep Satisfaction Prediction Network (DSPN), which models advertiser intent and satisfaction simultaneously. It employs a two-stage network structure where advertiser intent vector and satisfaction are jointly learned by considering the features of advertiser's action information and advertising performance indicators. Experiments on an Alibaba advertisement dataset and online evaluations show that our proposed DSPN outperforms state-of-the-art baselines and has stable performance in terms of AUC in the online environment. Further analyses show that DSPN not only predicts advertisers' satisfaction accurately but also learns an explainable advertiser intent, revealing the opportunities to optimize the advertising performance further.

show abstract

Deep Sequential Models for Task Satisfaction Prediction

Cited by 22 publications

References 42 publications

The Impact of Task Abandonment in Crowdsourcing

The Impact of Task Abandonment in Crowdsourcing

Neural Correlates of Realisation of Satisfaction in a Successful Search Process

A Deep Prediction Network for Understanding Advertiser Intent and Satisfaction

Contact Info

Product

Resources

About