[Invited Paper] TRECVid Semantic Indexing of Video: A 6-Year Retrospective

Abstract

Semantic indexing, or assigning semantic tags to video samples, is a key component for content-based access to video documents and collections. The Semantic Indexing task has been run at TRECVid from 2010 to 2015 with the support of NIST and the Quaero project. As with the previous High-Level Feature detection task which ran from 2002 to 2009, the semantic indexing task aims at evaluating methods and systems for detecting visual, auditory or multi-modal concepts in video shots. In addition to the main semantic indexing task, four secondary tasks were proposed namely the "localization" task, the "concept pair" task, the "no annotation" task, and the "progress" task. It attracted over 40 research teams during its running period. Both the increase in the volume of video data and in the number of target concepts favored the development of generic and scalable methods. Over 8 millions shots×concepts direct annotations plus over 20 millions indirect ones were produced by the participants and the Quaero project on a total of 800 hours of development data.Significant progress was accomplished during the period as this was accurately measured in the context of the progress task but also from some of the participants' contrast experiments. This paper describes the data, protocol and metrics used for the main and the secondary tasks, the results obtained and the main approaches used by participants.

Computer Vision for Lifelogging

“…For many years, within the context of the TRECVid video benchmarking activity, researchers struggled to achieve high enough accuracy for the classifiers, as well as large enough numbers of tags in order to be usable [39]. Then, in 2012, things changed with the significant improvements in recognition accuracy obtainable when deep learning networks were applied to this computer vision problem for classifying and tagging images, all led by the work of Geoffrey Hinton's team [40].…”

“…This is in contrast to current refinement methods which learn inter-concept relationships explicitly from training corpora and then apply these to test sets. Because acceptable detection results can be obtained for concept with enough training samples, as witnessed by TRECVid benchmark [39] and ImageNet competition [48], it is feasible to utilise detections with high accuracies to enhance overall multi-concept detections since the concepts are highly correlated.…”

“…Detecting activities and actions is much more complex than detecting objects, especially when the actions are spread over time, like "preparing a cup of tea" or "eating a meal". In 2016 the TRECVid activity introduced a task on automatic captioning of video [39] where the target videos to be captioned were social media videos from the Vine website, of up to 8 seconds each. For some videos, captioning was comparable to manual annotation but for many videos the automatic captions were poor because they lacked context, i.e.…”

“…The key organisational metaphor for the data here was as a diary with a chronological ordering of events on any given day. It is worth pointing out that an event-level browser or lifelog data can reduce many thousands of images per day to a more manageable set of events (typically about [30][31][32][33][34][35][36][37][38][39][40]. Such interfaces usually include drill-down functionality to enable the user to examiner some or all images that occur within one selected event.…”

Semantic Indexing of Wearable Camera Images

“…The idea of automatically assigning semantic concepts or tags to an image or video has been the subject of research for decades but more progress has been made within the last few years than in those previous decades [1]. The incorporation of deep learning into the process, coupled with the emergence of huge searchable image resources and training data means that automatic tagging of images is now offered by many websites like Aylien, IMAGGA, and others, as a commodity tagging service.…”