Hybrid feature-based analysis of video’s affective content using protagonist detection

Zhu, Yingying; Min, Tong; Jiang, Zhaoneng; Zhong, Sheng-hua

doi:10.1016/j.eswa.2019.03.017

Cited by 18 publications

(19 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, obtaining such dynamic labels by subjective reports could be labor-intensive and time consuming [74,84] and automatic tagging using internet-based crowdsourcing methods [85][86][87] or information from other modalities (e.g. video content analysis [88], face expressions [89] and peripheral physiological responses [90] during video watching) could provide feasible alternative options. For another, the lacking of standardized large-scale datasets has made the performance evaluation across different methods difficult.…”

Section: Decoding Affective State Accurately and Continuouslymentioning

confidence: 99%

Ten challenges for EEG-based affective computing

Chen

Wang

et al. 2019

Brain Science Advances

View full text Add to dashboard Cite

The emerging field of affective computing focuses on enhancing computers’ ability to understand and appropriately respond to people’s affective states in human-computer interactions, and has revealed significant potential for a wide spectrum of applications. Recently, the electroencephalography (EEG) based affective computing has gained increasing interest for its good balance between mechanistic exploration and real-world practical application. The present work reviewed ten theoretical and operational challenges for the existing affective computing researches from an interdisciplinary perspective of information technology, psychology, and neuroscience. On the theoretical side, we suggest that researchers should be well aware of the limitations of the commonly used emotion models, and be cautious about the widely accepted assumptions on EEG-emotion relationships as well as the transferability of findings based on different research paradigms. On the practical side, we propose several operational recommendations for the challenges about data collection, feature extraction, model implementation, online system design, as well as the potential ethical issues. The present review is expected to contribute to an improved understanding of EEG-based affective computing and promote further applications.

show abstract

Section: Decoding Affective State Accurately and Continuouslymentioning

confidence: 99%

Ten challenges for EEG-based affective computing

Chen

Wang

et al. 2019

Brain Science Advances

View full text Add to dashboard Cite

show abstract

“…In many cases, computations are conducted over each frame of the video, and the average values of the computational results of the overall video are considered as visual features. Specifically, the color-related features often contain the histogram and variance of color [20,107,177], the proportions of color [82,86], the number of white frame and fades [177], the grayness [20], darkness ratio, color energy [132,170], brightness ratio and saturation [85,86], etc. In addition, the differences of dark and light can be reflected by the lighting key, which is used to evoke emotions in video and draw the attention of viewers by creating an emotional atmosphere [82].…”

Section: Content-related Featuresmentioning

confidence: 99%

“…ZCR [86] is used to separate different types of audio signals, such as music, environmental sound and speech of human. Besides these frequent related features, audio flatness [177], spectral flux [177], delta spectrum magnitude, harmony [86,111,177], band energy ratio, spectral centroid [49,177], and spectral contrast [86] are also utilized.…”

Section: Content-related Featuresmentioning

confidence: 99%

“…Pre-trained model, SoundNet, is utilized to learn audio features. Because the expressive emotions of video are induced and communicated by the protagonist in video in many cases, the features of protagonist are extracted from the key frame by a pre-trained CNN and used in video affective analysis in [176,177]. In addition to the protagonist, other objects in each frame of video also give insights into emotional expression of video.…”

Section: Content-related Featuresmentioning

confidence: 99%

“…Decision-level fusion fuses several results of different classifiers, and the final results are computed according to the fusion methods. One way of feature-level fusion is implemented by feature accumulation or concatenation [14,147,177]. In [14], two feature vectors for visual and audio data are averaged as the global genre representations.…”

Section: Data Fusionmentioning

confidence: 99%

See 2 more Smart Citations

Affective Computing for Large-scale Heterogeneous Multimedia Data

Zhao

Wang

Soleymani

et al. 2019

ACM Trans. Multimedia Comput. Commun. Appl.

View full text Add to dashboard Cite

The wide popularity of digital photography and social networks has generated a rapidly growing volume of multimedia data (i.e., image, music, and video), resulting in a great demand for managing, retrieving, and understanding these data. Affective computing (AC) of these data can help to understand human behaviors and enable wide applications. In this article, we survey the state-of-the-art AC technologies comprehensively for large-scale heterogeneous multimedia data. We begin this survey by introducing the typical emotion representation models from psychology that are widely employed in AC. We briefly describe the available datasets for evaluating AC algorithms. We then summarize and compare the representative methods on AC of different multimedia types, i.e., images, music, videos, and multimodal data, with the focus on both handcrafted features-based methods and deep learning methods. Finally, we discuss some challenges and future directions for multimedia affective computing.

show abstract