Bridging the Ultimate Semantic Gap

Abstract

Semantic search in video is a novel and challenging problem in information and multimedia retrieval. Existing solutions are mainly limited to text matching, in which the query words are matched against the textual metadata generated by users. This paper presents a state-of-the-art system for event search without any textual metadata or example videos. The system relies on substantial video content understanding and allows for semantic search over a large collection of videos. The novelty and practicality is demonstrated by the evaluation in NIST TRECVID 2014, where the proposed system achieves the best performance. We share our observations and lessons in building such a stateof-the-art system, which may be instrumental in guiding the design of the future system for semantic search in video.

Interpretable Embedding for Ad-Hoc Video Search

“…To deal with out-of-vocabulary (OOV) problem, query expansion with ontology influencing [47] and webly-label learning by crawling online visual data [18] are commonly adopted. Despite numerous progress [8,19,35], automatic selection of concepts to capture query semantics as well as context remains highly difficult. Human intervention is often required in practice, for example, by manually removing undesired concepts after automatic matching [34] or by hand-picking query phrases that should be matched with concepts [45].…”

Semantic-Based Video Retrieval Survey

“…In [39], the authors propose a state of art system search engine for video event search without any submitted example videos to the query which is called zero-example or 0Ex. The system consists of video semantic indexing component, semantic query generation component, multimedia search, and pseudo-relevance.…”

“…It orders the event concept vector scores in descending order, constructing an exponential curve, so that then selects the first k concepts. This paper uses a large concept pool that consists of 13.488 semantic concepts, the MAP of the proposed system is slightly better than the other systems such as [39]. A real-time video retrieval can be applied to this approach because the concept detection scores are previously measured to the database of videos and the semantic similarity computing is fast.…”

Field Studies with Multimedia Big Data: Opportunities and Challenges (Extended Version)

“…The use of automatic detectors means that all data is included in the search, whether the user has supplied tags or not. The search engine would enable the user to prioritize the various filters to sort the data [as in 70,141]. (Alternatively, the user could start with a similarity search, if they already had some relevant examples on hand.)…”