Multimodal feature fusion for robust event detection in web videos

Wu, Shuang; Vitaladevuni, Shiv; Zhuang, Xinhua; Tsakalidis, Stavros; Park, Unsang; Prasad, Rohit; Natarajan, Premkumar

doi:10.1109/cvpr.2012.6247814

Cited by 146 publications

(127 citation statements)

References 32 publications

Supporting

Mentioning

126

Contrasting

Order By: Relevance

“…For video retrieval, significant research effort has been devoted in the form of event detection applied to the TRECVID MED collection. We refer the readers to an excellent state-of-the-art work of Natarajan et al [11] for more details. Video recognition has been an active research area in computer vision.…”

Section: Previous Workmentioning

confidence: 99%

Discovering Video Clusters from Visual Features and Noisy Tags

Vahdat

Zhou

Mori

2014

Computer Vision – ECCV 2014

View full text Add to dashboard Cite

Abstract. We present an algorithm for automatically clustering tagged videos. Collections of tagged videos are commonplace, however, it is not trivial to discover video clusters therein. Direct methods that operate on visual features ignore the regularly available, valuable source of tag information. Solely clustering videos on these tags is error-prone since the tags are typically noisy. To address these problems, we develop a structured model that considers the interaction between visual features, video tags and video clusters. We model tags from visual features, and correct noisy tags by checking visual appearance consistency. In the end, videos are clustered from the refined tags as well as the visual features. We learn the clustering through a max-margin framework, and demonstrate empirically that this algorithm can produce more accurate clustering results than baseline methods based on tags or visual features, or both. Further, qualitative results verify that the clustering results can discover sub-categories and more specific instances of a given video category.

show abstract

Section: Previous Workmentioning

confidence: 99%

Discovering Video Clusters from Visual Features and Noisy Tags

Vahdat

Zhou

Mori

2014

Computer Vision – ECCV 2014

View full text Add to dashboard Cite

show abstract

“…Distance from threshold This is a weighted averaging method [3] that dynamically adjusts the weights of each data type for each video clip based on how far the score is from its decision threshold. If the detection score is near the threshold, the correct decision is presumed to be somewhat uncertain, and a lower weight is assigned.…”

Section: Sparse Mixture Model (Smm)mentioning

confidence: 99%

“…In our approach, an event is modeled as a set of multiple bags-of-words, each based on a single data type. Partitioning the representation by data type permits the descriptors for each data type to be optimized independently (specific multimodal combinations of features, such as bimodal audiovisual features [3], can be considered a single data type within this architecture). The data types we used included both low-level features (visual appearance, motion, and audio) and higherlevel semantic concepts (visual concepts).…”

Section: Introductionmentioning

confidence: 99%

“…The most common approach is to apply late fusion methods [3,5,17] in which the results for each data type are combined by fusing the decision scores from multiple event classifiers. This is a straightforward way of using the information from all data types in proportion to their relative contribution to event detection on videos with widely diverse content.…”

Section: Introductionmentioning

confidence: 99%

“…Current approaches for MED [2][3][4][5][6][7] rely heavily on kernelbased classifier methods that use low-level features computed directly from the multimedia data. These classifiers learn a mapping between the computed features and the category of event that occurs in the video.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Evaluating multimedia features and fusion for example-based event detection

Myers

Nallapati

Hout

et al. 2013

Machine Vision and Applications

View full text Add to dashboard Cite

Multimedia event detection (MED) is a challenging problem because of the heterogeneous content and variable quality found in large collections of Internet videos. To study the value of multimedia features and fusion for representing and learning events from a set of example video clips, we created SESAME, a system for video SEarch with Speed and Accuracy for Multimedia Events. SESAME includes multiple bag-of-words event classifiers based on single data types: low-level visual, motion, and audio features; highlevel semantic visual concepts; and automatic speech recognition. Event detection performance was evaluated for each event classifier. The performance of low-level visual and motion features was improved by the use of difference coding. The accuracy of the visual concepts was nearly as strong as that of the low-level visual features. Experiments with a number of fusion methods for combining the event detection scores from these classifiers revealed that simple fusion methods, such as arithmetic mean, perform as well as or better than other, more complex fusion methods. SESAME's

show abstract

Web video classification with visual and contextual semantics

Afzal

Shah

Tufail

2019

Int J Communication

View full text Add to dashboard Cite

Summary On the social Web, the amount of video content either originated from wireless devices or previously received from media servers has increased enormously in the recent years. The astounding growth of Web videos has stimulated researchers to propose new strategies to organize them into their respective categories. Because of complex ontology and large variation in content and quality of Web videos, it is difficult to get sufficient, precisely labeled training data, which causes hindrance in automatic video classification. In this paper, we propose a novel content‐ and context‐based Web video classification framework by rendering external support through category discriminative terms (CDTs) and semantic relatedness measure (SRM). Mainly, a three‐step framework is proposed. Firstly, content‐based video classification is proposed, where twofold use of high‐level concept detectors is leveraged to classify Web videos. Initially, category classifiers induced from VIREO‐374 detectors are trained to classify Web videos, and then concept detectors with high confidence for each video are mapped to CDT through SRM‐assisted semantic content fusion function to further boost the category classifiers, which intuitively provide a more robust measure for Web video classification. Secondly, a context‐based video classification is proposed, where twofold use of contextual information is also harnessed. Initially, cosine similarity and then semantic similarity are measured between text features of each video and CDT through vector space model (VSM)‐ and SRM‐assisted semantic context fusion function, respectively. Finally, classification results from content and context are fused to compensate for the shortcomings of each other, which enhance the video classification performance. Experiments on large‐scale video dataset validate the effectiveness of the proposed solution.

show abstract

Multimodal feature fusion for robust event detection in web videos

Cited by 146 publications

References 32 publications

Discovering Video Clusters from Visual Features and Noisy Tags

Discovering Video Clusters from Visual Features and Noisy Tags

Evaluating multimedia features and fusion for example-based event detection

Web video classification with visual and contextual semantics

Contact Info

Product

Resources

About