Image ranking with implicit feedback from eye movements

Abstract

In order to help users navigate an image search system, one could provide explicit information on a small set of images as to which of them are relevant or not to their task. These rankings are learned in order to present a user with a new set of images that are relevant to their task. Requiring such explicit information may not be feasible in a number of cases, we consider the setting where the user provides implicit feedback, eye movements, to assist when performing such a task. This paper explores the idea of implicitly incorporating eye movement features in an image ranking task where only images are available during testing. Previous work had demonstrated that combining eye movement and image features improved on the retrieval accuracy when compared to using each of the sources independently. Despite these encouraging results the proposed approach is unrealistic as no eye movements will be presented a-priori for new images (i.e. only after the ranked images are presented would one be able to measure a user's eye movements on them). We propose a novel search methodology which combines image features together with implicit feedback from users' eye movements in a tensor ranking Support Vector Machine and show that it is possible to extract the individual source-specific weight vectors. Furthermore, we demonstrate that the decomposed image weight vector is able to construct a new image-based semantic space that outperforms the retrieval accuracy than when solely using the image-features.

Multimedia implicit tagging using EEG signals

“…Implicit tagging has been used in image annotation, video highlight detection, topical relevance detection and retrieval result re-ranking. The existing literature can be divided into two categories, one dealing with using emotional reactions to tag the content with the expressed emotion, e.g., laughter detection for hilarity [16], and the second group of studies using the spontaneous reactions for information retrieval or search results, e.g., eye gaze for relevance feedback [17].…”

Eye-tracker based part-image selection for image retrieval

“…An image retrieval system using a combination of eye tracking and visual features from the image was presented in [19]. Researchers have also proposed using eye tracking as a relevance feedback mechanism for image retrieval systems [20,21] and on applying image re-ranking based on such relevance feedback [22]. In this work, we extend the stateof-the-art in the direction of part-image search and retrieval using eye tracking technology.…”

Human-centered implicit tagging: Overview and perspectives

“…Implicit tagging has been used in image annotation, video highlight detection, topical relevance detection and retrieval result re-ranking. The existing literature can be divided into two categories, one dealing with using emotional reactions to tag the content with the expressed emotion, e.g., laughter detection for hilarity [5], and the second group of studies using the spontaneous reactions for information retrieval or search results, e.g., eye gaze for relevance feedback [22]. A summary of the recent relevant literature on this topic is given in Table I. There have been also studies using unnimodal or multimodal approaches for detecting the behavioral or emotional responses to multimedia [21], [23], [24], [20], [6].…”