A Multimodal Variational Encoder-Decoder Framework for Micro-video Popularity Prediction

Xie, Jing; Zhu, Ye; Zhang, Zhibin; Peng, Jian; Yi, Jianxin; Hu, Yaosi; Liu, Hongyi; Chen, Zhenzhong

doi:10.1145/3366423.3380004

Cited by 29 publications

(15 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They achieve state-of-the-art results for solving particular machine learning problems. It is practically impossible to analyze all of them, but a significant number of them, at one step or another, use the classical concatenation of multimodal vectors [51]- [53], without a deep examination of unique dependencies between them. Nevertheless, there are other models proposing smarter modality aggregation, such as the Contrastive Multimodal Fusion method [54], showing there is growing interest in the ML community for nontrivial multimodal fusion.…”

Section: B Brain-inspired Multimodal Frameworkmentioning

confidence: 99%

A unified software/hardware scalable architecture for brain-inspired computing based on self-organizing neural models

Muliukov,

Rodriguez,

Miramond

et al. 2022

Preprint

View full text Add to dashboard Cite

The field of artificial intelligence has significantly advanced over the past decades, inspired by discoveries from the fields of biology and neuroscience. The idea of this work is inspired by the process of self-organization of cortical areas in the human brain from both afferent and lateral/internal connections. In this work, we develop an original brain-inspired neural model associating Self-Organizing Maps (SOM) and Hebbian learning in the Reentrant SOM (ReSOM) model. The framework is applied to multimodal classification problems. Compared to existing methods based on unsupervised learning with post-labeling, the model enhances the state-of-the-art results. This work also demonstrates the distributed and scalable nature of the model through both simulation results and hardware execution on a dedicated FPGA-based platform named SCALP (Self-configurable 3D Cellular Adaptive Platform). SCALP boards can be interconnected in a modular way to support the structure of the neural model. Such a unified software and hardware approach enables the processing to be scaled and allows information from several modalities to be merged dynamically. The deployment on hardware boards provides performance results of parallel execution on several devices, with the communication between each board through dedicated serial links. The proposed unified architecture, composed of the ReSOM model and the SCALP hardware platform, demonstrates a significant increase in accuracy thanks to multimodal association, and a good trade-off between latency and power consumption compared to a centralized GPU implementation.

show abstract

Section: B Brain-inspired Multimodal Frameworkmentioning

confidence: 99%

A unified software/hardware scalable architecture for brain-inspired computing based on self-organizing neural models

Muliukov,

Rodriguez,

Miramond

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Researchers from multimedia and data mining communities have done a considerable number of works to predict the popularity of online videos. Most of these works intend to predict the popularity of videos from online video or social media websites, such as YouTube [45,52,59], Vine [10,37,48,61], Facebook [4,5], Kuaishou [65], etc. Multi-modal data including popularity evolution pattern and social media, visual, acoustic, textual, geography features are exploited for prediction.…”

Section: Related Workmentioning

confidence: 99%

“…Chen et al [10] propose to predict the popularity of microvideos from Vine using a transductive model, in which social, visual, acoustic and textual features are taken as the input. Following this work, methods like variational encoder-decoder [61] and feature-discrimination transductive model [48] are explored to better capture popularity-related information from the same features. Visual cues are combined with early evolution patterns in [52] to train an support vector regressor (SVR) for popularity prediction.…”

Section: Related Workmentioning

confidence: 99%

Will You Ever Become Popular? Learning to Predict Virality of Dance Clips

Wang

Weng

et al. 2022

ACM Trans. Multimedia Comput. Commun. Appl.

View full text Add to dashboard Cite

Dance challenges are going viral in video communities like TikTok nowadays. Once a challenge becomes popular, thousands of short-form videos will be uploaded within a couple of days. Therefore, virality prediction from dance challenges is of great commercial value and has a wide range of applications, such as smart recommendation and popularity promotion. In this article, a novel multi-modal framework that integrates skeletal, holistic appearance, facial and scenic cues is proposed for comprehensive dance virality prediction. To model body movements, we propose a pyramidal skeleton graph convolutional network (PSGCN) that hierarchically refines spatio-temporal skeleton graphs. Meanwhile, we introduce a relational temporal convolutional network (RTCN) to exploit appearance dynamics with non-local temporal relations. An attentive fusion approach is finally proposed to adaptively aggregate predictions from different modalities. To validate our method, we introduce a large-scale viral dance video (VDV) dataset, which contains over 4,000 dance clips of eight viral dance challenges. Extensive experiments on the VDV dataset well demonstrate the effectiveness of our approach. Furthermore, we show that short video applications such as multi-dimensional recommendation and action feedback can be derived from our model.

show abstract

“…For example, TLRMVR [8] proposes a novel low-rank multiview embedding learning method to predict the popularity of microvideo. MMVED [9] combines multiple features (image frame, acoustic, and textual info) and considers the randomness for the mapping from data to popularity. Although these approaches are able to achieve efficient prediction, they usually aim at the static complete file and need to parse entire image frame, which is impractical for live streaming.…”

Section: Relate Workmentioning

confidence: 99%

“…These approaches usually fail to address the rationality of MEC application providers or have different objective functions, leading to improper results in cost optimization for live streaming. Some works optimize the resource allocation by predicting the popularity of contents [8][9][10], considering that a few popular videos usually contribute to most of the bandwidth consumption [11]. However, suffering from the similar reason that these models usually focus on the prediction of static contents, it is still hard to meet the real-time requirement of live streaming.…”

Section: Introductionmentioning

confidence: 99%

Popularity-Guided Cost Optimization for Live Streaming in Mobile Edge Computing

Zhu

Chen

et al. 2022

Wireless Communications and Mobile Computing

View full text Add to dashboard Cite

Live streaming service usually delivers the content in mobile edge computing (MEC) to reduce the network latency and save the backhaul capacity. Considering the limited resources, it is necessary that MEC servers collaborate with each other and form an overlay to realize more efficient delivery. The critical challenge is how to optimize the topology among the servers and allocate the link capacity so that the cost will be lower with delay constraints. Previous approaches rarely consider server collaborations for live streaming service, and the scheduling delay is usually ignored in MEC, leading to suboptimal performances. In this paper, we propose a popularity-guided overlay model which takes the scheduling delay into consideration and utilizes MEC collaboration to achieve efficient live streaming service. The links and servers are shared among all channel streams and each stream is pushed from cloud servers to MEC servers via the trees. Considering the optimization problem is NP-hard, we propose an effective optimization framework called cost optimization for live streaming (COLS) to predict the channel popularity by a LSTM model with multiscale input data. Finally, we compute topology graph by greedy scheme and allocate the capacity with convex programming. Experimental results show that the proposed approach achieves higher prediction accuracy, reducing the capacity cost by more than 40% with an acceptable delay compared with state-of-the-art schemes.

show abstract

A Multimodal Variational Encoder-Decoder Framework for Micro-video Popularity Prediction

Cited by 29 publications

References 12 publications

A unified software/hardware scalable architecture for brain-inspired computing based on self-organizing neural models

A unified software/hardware scalable architecture for brain-inspired computing based on self-organizing neural models

Will You Ever Become Popular? Learning to Predict Virality of Dance Clips

Popularity-Guided Cost Optimization for Live Streaming in Mobile Edge Computing

Contact Info

Product

Resources

About