Audiovisual Classification of Group Emotion Valence Using Activity Recognition Networks

Pinto, João Ribeiro; Gonçalves, Tiago; Pinto, Carolina; Sanhudo, Luís; Fonseca, Joaquim; Gonçalves, Filipe; Carvalho, P.; Cardoso, Jaime S.

doi:10.1109/ipas50080.2020.9334943

Cited by 13 publications

(5 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Details Modality Accuracy, % VGG13 + VGG16 + ResNet [4] Ensemble video 59.42 Hu et al [25] Ensemble audio, video 59.01 Four face recognition CNNs+STAT+FFT [18] Ensemble video 56.7 Noisy student with iterative training [26] Single model video 55.17 Noisy student w/o iterative training [26] Single model video 52.49 DenseNet-161 [27] Single model video 51.44 Face attention network (FAN) [28] Single model video 51.18 LBP-TOP (baseline) [11] Single Method Details Modality Accuracy, % Hybrid Networks [5] Ensemble audio, video 74.28 K-injection network [29] Ensemble audio, video 66.19 DenseNet-121 (FER+) [5] Single model video 64.75 Activity Recognition Networks [30] sults, the accuracy of MobileNet features here is 2% higher when compared to EfficientNet, so that we could claim that both models have their advantages in various emotion recognition tasks.…”

Section: Methodsmentioning

confidence: 99%

Facial expression and attributes recognition based on multi-task learning of lightweight neural networks

Savchenko

2021

Preprint

View full text Add to dashboard Cite

In this paper, we examine the multi-task training of lightweight convolutional neural networks for face identification and classification of facial attributes (age, gender, ethnicity) trained on cropped faces without margins. It is shown that it is still necessary to fine-tune these networks in order to predict facial expressions. Several models are presented based on MobileNet, EfficientNet and RexNet architectures. It was experimentally demonstrated that our models are characterized by the state-of-the-art emotion classification accuracy on AffectNet dataset and near state-ofthe-art results in age, gender and race recognition for UTKFace dataset. Moreover, it is shown that the usage of our neural network as a feature extractor of facial regions in video frames and concatenation of several statistical functions (mean, max, etc.) leads to 4.5% higher accuracy than the previously known state-of-the-art single models for AFEW and VGAF datasets from the EmotiW challenges. The models and source code are publicly available at https://github.com/HSE-asavchenko/ face-emotion-recognition.

show abstract

Section: Methodsmentioning

confidence: 99%

Facial expression and attributes recognition based on multi-task learning of lightweight neural networks

Savchenko

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Classifying human actions and activities is a challenging topic that benefited greatly from improvements in computational capabilities and neural networks. Action recognition and activity recognition are often used interchangeably [70]. This task is fundamental for scene understanding to capture object interactions.…”

Section: E Action Recognitionmentioning

confidence: 99%

From a Visual Scene to a Virtual Representation: A Cross-Domain Review

et al. 2023

Self Cite

View full text Add to dashboard Cite

The widespread use of smartphones and other low-cost equipment as recording devices, the massive growth in bandwidth, and the ever-growing demand for new applications with enhanced capabilities, made visual data a must in several scenarios, including surveillance, sports, retail, entertainment, and intelligent vehicles. Despite significant advances in analyzing and extracting data from images and video, there is a lack of solutions able to analyze and semantically describe the information in the visual scene so that it can be efficiently used and repurposed. Scientific contributions have focused on individual aspects or addressing specific problems and application areas, and no cross-domain solution is available to implement a complete system that enables information passing between cross-cutting algorithms. This paper analyses the problem from an end-to-end perspective, i.e., from the visual scene analysis to the representation of information in a virtual environment, including how the extracted data can be described and stored. A simple processing pipeline is introduced to set up a structure for discussing challenges and opportunities in different steps of the entire process, allowing to identify current gaps in the literature. The work reviews various technologies specifically from the perspective of their applicability to an endto-end pipeline for scene analysis and synthesis, along with an extensive analysis of datasets for relevant tasks.

show abstract

“…Beyond detecting violence, the recognition of occupant's activity using different modalities has also been explored, specifically using audiovisual features [1]. Audio features appeared from applying this type of feature for the classification of group emotion [40]. This work focused primarily on the available hardware and energy consumption constraints associated with implementing an activity recognition system in a vehicle.…”

Section: In-vehicle Occupant Activity Recognitionmentioning

confidence: 99%

“…The baseline model used in this work is based on the model proposed in [40], presented in Figure 1. It uses RGB data for recognizing actions, and it has achieved results comparable to state-of-the-art methodologies.…”

Section: A Baselinementioning

confidence: 99%

See 1 more Smart Citation

Toward Vehicle Occupant-Invariant Models for Activity Characterization

Capozzi

Barbosa

Pinto³

et al. 2022

IEEE Access

Self Cite

View full text Add to dashboard Cite

With the advent of self-driving cars and the push by large companies into fully driverless transportation services, monitoring passenger behaviour in vehicles is becoming increasingly important for several reasons, such as ensuring safety and comfort. Although several human action recognition (HAR) methods have been proposed, developing a true HAR system remains a very challenging task. If the dataset used to train a model contains a small number of actors, the model can become biased towards these actors and their unique characteristics. This can cause the model to generalise poorly when confronted with new actors performing the same actions. This limitation is particularly acute when developing models to characterise the activities of vehicle occupants, for which data sets are short and scarce. In this study, we describe and evaluate three different methods that aim to address this actor bias and assess their performance in detecting in-vehicle violence. These methods work by removing specific information about the actor from the model's features during training or by using data that is independent of the actor, such as information about body posture. The experimental results show improvements over the baseline model when evaluated with real data.

show abstract

Audiovisual Classification of Group Emotion Valence Using Activity Recognition Networks

Cited by 13 publications

References 21 publications

Facial expression and attributes recognition based on multi-task learning of lightweight neural networks

Facial expression and attributes recognition based on multi-task learning of lightweight neural networks

From a Visual Scene to a Virtual Representation: A Cross-Domain Review

Toward Vehicle Occupant-Invariant Models for Activity Characterization

Contact Info

Product

Resources

About