Automatic Transformation of a Video Using Multimodal Information for an Engaging Exploration Experience

Abstract

Exploring the content of a video is typically inefficient due to the linear streamed nature of its media and the lack of interactivity. While different approaches have been proposed for enhancing the exploration experience of video content, the general view of video content has remained basically the same, that is, a continuous stream of images. It is our contention that such a conservative view on video limits its potential value as a content source. This paper presents An Alternative Representation of Video via feature Extraction (RAAVE), a novel approach to transform videos from a linear stream of content into an adaptive interactive multimedia document and thereby enhance the exploration potential of video content by providing a more engaging user experience. We explore the idea of viewing video as a diverse multimedia content source, opening new opportunities and applications to explore and consume video content. A modular framework and algorithm for the representation engine and template collection is described. The representation engine based approach is evaluated through development of a prototype system grounded on the design of the proposed approach, allowing users to perform multiple content exploration tasks within a video. The evaluation demonstrated RAAVE’s ability to provide users with a more engaging, efficient and effective experience than a typical multimedia player while performing video exploration tasks.

MTR-SAM: Visual Multimodal Text Recognition and Sentiment Analysis in Public Opinion Analysis on the Internet

“…With technological developments in single-modal AI technology, there is a developing desire among academics for multimodal information technology. Methods for multimodal text recognition attempt to combine visual feature information such as image, video, audio, etc., first through translation [17] and alignment [18] of modal information. The model is then pretrained to facilitate cross-modal interaction between features to learn about the concealed wealth of information between modalities.…”

A Comparative analysis of Video Summarization techniques for different domains

“…Color, Motion, Low-level Descriptor, Clustering based approaches, [21,28,29,32] Lecture Color , Motion and Clustering based approach [23,30,29,32] Surveillance Graph and Event-based approaches [9] Wild Life Color and Clustering based approaches Motion-based Rugby and Soccer Sports videos Keyframe Subjective [16] Deep Learning Based YoutTube videos ,SumMe and TVSum Dataset…”