Extended conceptual feedback for semantic multimedia indexing

Abstract

International audienceIn this paper, we consider the problem of automatically detecting a large number of visual concepts in images or video shots. State of the art systems generally involve feature (descriptor) extraction, classification (supervised learning) and fusion when several descriptors and/or classifiers are used. Though direct multi-label approaches are considered in some works, detection scores are often computed independently for each target concept. We propose a method that we call "conceptual feedback" which implicitly takes into account the relations between concepts to improve the overall concepts detection performance. A conceptual descriptor is built from the system's output scores and fed back by adding it to the pool of already available descriptors. Our proposal can be iterated several times. Moreover, we propose three extensions of our method. Firstly, a weighting of the conceptual dimensions is performed to give more importance to concepts which are more correlated to the target concept. Secondly, an explicit selection of a set of concepts that are semantically or statically related to the target concept is introduced. For video indexing, we propose a third extension which integrates the temporal dimension in the feedback process by taking into account simultaneously the conceptual and the temporal dimensions to build the high-level descriptor. Our proposals have been evaluated in the context of the TRECVid 2012 semantic indexing task involving the detection of 346 visual or multi-modal concepts. Overall, combined with temporal re-scoring, the proposed method increased the global system performance (MAP) from 0.2613 to 0.3082 (+ 17.9 % of relative improvement) while the temporal re-scoring alone increased it only from 0.2613 to 0.2691 (+ 3.0 %)

Learned features versus engineered features for semantic video indexing

“…Finally, two other improvement techniques were tried: conceptual re-scoring and use of an uploader model. Conceptual re-scoring is different from conceptual feedback, it is similar to temporal re-scoring but it exploits the semantic similarity between concepts instead of the temporal closeness between video shots [14]. It did not prove useful probably because, even if based on a different method, it captures the same type of information as the conceptual feedback done previously.…”

“…• Concepts features corresponding to the conceptual feedback approach [14] applied two times. These are were originally designed for being used with engineered descriptors but they acn also include other semantic descriptors; here they have been computed including the Xerox semantic descriptors.…”

“…It is also greater for semantic features than for quasi-semantic ones (corresponding to internal layers). It also appears smaller for conceptual feedback features but this is due to the fact that the feedback features were computed from scores that were already re-scored as it works better like this [14]. Figure 4 shows the performance gain brought by the successive fusion of descriptors of increasing performance.…”

“…In all these works and many other similar ones, the semantic features are learned on completely different collections and generally for concepts or categories different from those searched for on the target collection. Hamadi et al [14] used the approach using the same collection and the same concepts both for the semantic feature training and for their use in a further classification step. In this variant, called "conceptual feedback", a given target concept is learned both from the "low-level" features and from the detection scores of the other target concepts also learned from the same low-level features (the training of the semantic features has to be done by crossvalidation within the training set so that it can be used for the second training step both on the training and test sets).…”

Temporal re-scoring vs. temporal descriptors for semantic indexing of videos

“…The operation continues until no shot with a significant correlation is found. In [14], the authors exploit the idea presented in [8], [9] and propose another approach that consists of generating a descriptor by performing an early fusion of high-level descriptors of shots belonging to a temporal window centered on the current shot. They achieved very interesting results and enhanced a good baseline system.…”