Multimedia event detection using visual concept signatures

Abstract

Multimedia Event Detection (MED) is a multimedia retrieval task with the goal of finding videos of a particular event in a large-scale Internet video archive, given example videos and text descriptions. In this paper, we mainly focus on an 'ad-hoc' scenario in MED where we do not use any example video. We aim to retrieve test videos based on their visual semantics using a Visual Concept Signature (VCS) generated for each event only derived from the event description provided as the query. Visual semantics are described using the Semantic INdexing (SIN) feature which represents the likelihood of predefined visual concepts in a video. To generate a VCS for an event, we project the given event description to a visual concept list using the proposed textual semantic similarity. Exploring SIN feature properties, we harmonize the generated visual concept signature and the SIN feature to improve retrieval performance. We conduct different experiments to assess the quality of generated visual concept signatures with respect to human expectation, and in the context of the MED task to retrieve the SIN feature of videos in the test dataset when we have no or only very few training videos.

Content-Based Multimedia Retrieval Using Feature Correlation Clustering and Fusion

“…In addition, it is also important to legitimately demonstrate the performance in multimedia retrieval by comparing with other related works, which also explore multi-modalities. For example, many researchers explored the cross-modal relationship by applying canonical correlation analysis to better perform in multimedia retrieval (Rasiwasia et al, 2010;Zhang & Liu, 2012), face recognition (Guam, Zhang, Luo, & Lan, 2012), event detection (Younessian, Quinn, Mitamura, & Hauptmann, 2013), etc. In addition, besides positive correlation, Zhai et al also pointed out the importance of capturing negative cross-modality correlation since it can provide exclusive information.…”

A framework for associated news story retrieval

“…Particularly, in video retrieval tasks using only text-based query, we can retrieve visual concepts in a video using visual concept signature generated based on a given text query. For instance in [123], visual concept signature idea, explained in Section 6.2.1, was used to tackle the ad-hoc Multimedia Event Detection task. Multimedia Event Detection (MED) is a multimedia retrieval task with the goal of finding videos of a particular event (e.g.…”

“…"getting a vehicle unstuck", "wedding", "making a sandwich", etc) in a large-scale internet video archive, given text descriptions of events. In [123], the test videos were retrieved based on their visual semantics using a Visual Concept Signature (VCS) generated for each event only derived from the event description provided as the query. Visual semantics are described using the Semantic Indexing (SIN) feature which represents the likelihood of predefined visual concepts in a video, similar…”

“…Through a set of experiments on the MED task [130], the effectiveness of the VCS and the harmonizing step to retrieve the test videos were shown. More details about the proposed approach and the experiment results can be found in [123].…”