Identifying Objects in Images from Analyzing the Users’ Gaze Movements for Provided Tags

Abstract

Abstract. Millions of users share, tag, and search for images on social media platforms and social networking sites today. Annotating and searching for specific image regions, however, is still very hard. Assuming that eye tracking will be a common input device in the near future in notebooks equipped with cameras and mobile devices like iPads, it is possible to implicitly gain information about images and image regions from these users' gaze movements. In this paper, we investigate the principle idea of finding specific objects shown in images by looking at the users' gaze path information only. We have analyzed 547 gaze paths from 20 subjects viewing different image-tag-pairs with the task to decide if the tag presented is actually found in the image or not. By analyzing the gaze paths, we are able to correctly identify 67% of the image regions and significantly outperform two baselines. In addition, we have investigated if different regions of the same image can be differentiated by the gaze information. Here, we are able to correctly identify two different regions in the same image with an accuracy of 38%.

Can You See It? Two Novel Eye-Tracking-Based Measures for Assigning Tags to Image Regions

“…Example eye tracking measures are the fixationCount, a standard measure which counts the number of fixations on an region and the lastFixationDuration, which sums up the duration of the last fixation on an image region. We have compared 13 fixation measures with respect to their ability to identify a concrete image region for a tag t given to the users [14]. Derived from the results of this work, we use the measure lastFixationDuration, which has delivered the best results.…”

“…Different kinds of heat maps can be created based on different measures, e.g., a fixationCount or a absoluteDuration heat map [5]. As the last lastFixationDuration was the best measurement for the region identification in our previous work [14], we use this measure as basis for our approach. A radius rd has to be defined for the creation of a heat map.…”

“…An illustration of this thresholding is presented in Figure 7. Baseline Approach: Initially, we had investigated a random baseline approach as used in our previous work [14], which is randomly selecting one segment of an automatically segmented image as favorite region. As the results of this baseline were very weak, we decided to improve the baseline approach by taking into account the position of the segments in the image in two different ways.…”

“…The eye tracking data for our investigations is taken from a controlled experiment conducted with 30 subjects each viewing 51 social media images with given tags. The experiment is presented in [14]. First, the subjects where shown a specific tag.…”

“…General identification of image regions showing specific objects like it is aimed in this work is not conducted. In a previous work [14], we have investigated the possibilities to assign tags to image regions, where these regions were manually labeled with hand-drawn polygons. Gaze paths of users looking at the images were analyzed by 13 different fixation measures to calculate the assignment.…”