Multimodal Video Indexing (MVI): A New Method Based on Machine Learning and Semi-Automatic Annotation on Large Video Collections

Abstract

Indexing video by the concept is one of the most appropriate solutions for such problems. It is based on an association between a concept and its corresponding visual sound, or textual features. This kind of association is not a trivial task. It requires knowledge about the concept and its context. In this paper, we investigate a new concept detection approach to improve the performance of content-based multimedia documents retrieval systems. To achieve this goal, we are going to tackle the problem from different plans and make four contributions at various stages of the indexing process. We propose a new method for multimodal indexation based on (i) a new weakly supervised semi-automatic method based on the genetic algorithm (ii) the detection of concepts from the text in the videos (iii) the enrichment of the basic concepts thanks to the usage of our method DCM. Subsequently, the semantic and enriched concepts allow a better multimodal indexation and the construction of an ontology. Finally, the different contributions are tested and evaluated on a large dataset (TRECVID 2015).