Rich features for perceptual quality assessment of UGC videos

Wang, Yilin; Talebi, Hossein; Yim, Joong Gon; Birkbeck, Neil; Adsumilli, Balu; Milanfar, Peyman; Yang, Feng

doi:10.1109/cvpr46437.2021.01323

Cited by 67 publications

(37 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We observe that models using GRU and Transformer are both inferior to the proposed model, which means that the MLP module is enough to regress the quality-aware features to quality scores though it is very simple. This conclusion is also consistent with [29]. The reason is that the proposed model and the model in [29] calculate the chunk-level quality score and the effect of adjacent frames are considered in the quality-aware features (i.e.…”

Section: Quality Regression Modulesupporting

confidence: 84%

“…This conclusion is also consistent with [29]. The reason is that the proposed model and the model in [29] calculate the chunk-level quality score and the effect of adjacent frames are considered in the quality-aware features (i.e. motion features), while other VQA models [13] [12] calculate the frame-level quality scores, which may need to consider the effect of adjacent frames in the quality regression module.…”

Section: Quality Regression Modulesupporting

confidence: 77%

“…have greatly promoted the development of UGC VQA. Several deep learning based NR VQA models [12,13,29,32,35] have been proposed to solve some challenges mentioned above and achieve pretty good performance. However, there are still some problems that need to be addressed.…”

Section: Introductionmentioning

confidence: 99%

“…However, there are still some problems that need to be addressed. First, the previous studies either use the image recognition model [13] [35] or the pretrained image quality assessment (IQA) models [29][32] [12] to extract frame-level features, which lacks an end-to-end learning method to learn the quality-aware spatial feature representation from raw pixels of video frames. Specifically, the features extracted by the image recognition model are sensitive to semantic information but insensitive to low-level visual distortions, which widely exist in the UGC videos.…”

Section: Introductionmentioning

confidence: 99%

“…[35] propose a local-to-global region-based NR VQA model that combines the spatial features extracted from a 2D-CNN model and the spatial-temporal features from a 3D-CNN network. Wang et al[29] propose a feature-rich VQA model for UGC videos, which measures the quality from three aspects, compression level, video content, and distortion type and each aspect is evaluated by an individual neural network. Xu et al[32] first extract the spatial feature of the video frame from a pre-trained IQA model and use the graph convolution to extract and enhance these features, then extract motion information from the optical flow domain, and finally integrated the spatial feature and motion information via a bidirectional long short-term memory network.…”

mentioning

confidence: 99%

See 4 more Smart Citations

A Deep Learning based No-reference Quality Assessment Model for UGC Videos

Sun¹,

Min²,

Lü³

et al. 2022

Preprint

View full text Add to dashboard Cite

Quality assessment for User Generated Content (UGC) videos plays an important role in ensuring the viewing experience of end-users. Previous UGC video quality assessment (VQA) studies either use the image recognition model or the image quality assessment (IQA) models to extract frame-level features of UGC videos for quality regression, which are regarded as the sub-optimal solutions because of the domain shifts between these tasks and the UGC VQA task. In this paper, we propose a very simple but effective UGC VQA model, which tries to address this problem by training an end-to-end spatial feature extraction network to directly learn the quality-aware spatial feature representation from raw pixels of the video frames. We also extract the motion features to measure the temporal-related distortions that the spatial features cannot model. The proposed model utilizes very sparse frames to extract spatial features and dense frames (i.e. the video chunk) with a very low spatial resolution to extract motion features, which thereby has low computational complexity. With the better quality-aware features, we only use the simple multilayer perception layer (MLP) network to regress them into the chunk-level quality scores, and then the temporal average pooling strategy is adopted to obtain the video-level quality score. We further introduce a multi-scale quality fusion strategy to solve the problem of VQA across different spatial resolutions, where the multi-scale weights are obtained from the contrast sensitivity function of the human visual system. The experimental results show that the proposed model achieves the best performance on five popular UGC VQA databases, which demonstrates the effectiveness of the proposed model. The code will be publicly available.

show abstract

Section: Quality Regression Modulesupporting

confidence: 84%

Section: Quality Regression Modulesupporting

confidence: 77%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

A Deep Learning based No-reference Quality Assessment Model for UGC Videos

Sun¹,

Min²,

Lü³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

It pays to diversify: Effect of bullet‐screen comment diversity on payment

Zhang,

Ma,

Chen

2024

Psychology and Marketing

View full text Add to dashboard Cite

Bullet‐screen comments (BSCs), a type of novel user‐generated content for online videos, have been rapidly adopted in recent years. Previous research has focused on consumers' responses to the presence and quantity of BSCs but overlooked their contents. We examine how one characteristic feature of BSCs' content—lexical diversity—affects users' payments. A large‐scale real‐world behavioral data set analysis and three lab studies show that high (vs. low) lexical diversity of BSCs increases viewers' payments in online video streaming contexts (e.g., online videos and live streaming). We further reveal that the positive effect of BSC diversity on payment is mediated by perceived social presence and information richness. We also demonstrate the moderating role of viewers' BSC usage level (heavy vs. light), such that the effect of BSC diversity on payment is significant for heavy BSC users, but diminishes for light users.

show abstract

MaxViT: Multi-axis Vision Transformer

Talebi

Zhang

et al. 2022

Lecture Notes in Computer Science

Self Cite

325

View full text Add to dashboard Cite

Image resizing operation is a fundamental preprocessing module in modern computer vision. Throughout the deep learning revolution, researchers have overlooked the potential of alternative resizing methods beyond the commonly used resizers that are readily available, such as nearestneighbors, bilinear, and bicubic. The key question of our interest is whether the front-end resizer affects the performance of deep vision models? In this paper, we present an extremely lightweight multilayer Laplacian resizer with only a handful of trainable parameters, dubbed MULLER resizer. MULLER has a bandpass nature in that it learns to boost details in certain frequency subbands that benefit the downstream recognition models. We show that MULLER can be easily plugged into various training pipelines, and it effectively boosts the performance of the underlying vision task with little to no extra cost. Specifically, we select a state-of-the-art vision Transformer, MaxViT [50], as the baseline, and show that, if trained with MULLER, MaxViT gains up to 0.6% top-1 accuracy, and meanwhile enjoys 36% inference cost saving to achieve similar top-1 accuracy on ImageNet-1k, as compared to the standard training scheme. Notably, MULLER's performance also scales with model size and training data size such as ImageNet-21k and JFT, and it is widely applicable to multiple vision tasks, including image classification, object detection and segmentation, as well as image quality assessment.

show abstract

Rich features for perceptual quality assessment of UGC videos

Cited by 67 publications

References 28 publications

A Deep Learning based No-reference Quality Assessment Model for UGC Videos

A Deep Learning based No-reference Quality Assessment Model for UGC Videos

It pays to diversify: Effect of bullet‐screen comment diversity on payment

MaxViT: Multi-axis Vision Transformer

Contact Info

Product

Resources

About