Hierarchical Committee of Deep CNNs with Exponentially-Weighted Decision Fusion for Static Facial Expression Recognition

Kim, Bo-Kyeong; Lee, Hwaran; Roh, Jihyeon; Lee, Sang Yup

doi:10.1145/2818346.2830590

Cited by 99 publications

(52 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, more recently deep and recurrent neural networks have shown promising capabilities, outperforming previous results in dealing with rich multimodal data in areas like facial expression recognition (Kim et al 2015) or speech recognition, both in lab settings and even in the wild (Dhall et al 2015). …”

Section: Related Workmentioning

confidence: 99%

“…In terms of data processing, the initial forays into MMLA used relatively simple machine learning algorithms to build models of the phenomena under study (Ochoa et al, 2013). However, more recently, deep and recurrent neural networks (RNNs) have shown promising capabilities, outperforming previous results in dealing with rich multimodal data in areas such as facial expression recognition (Kim, Lee, Roh, & Lee, 2015) or speech recognition, both in lab settings and even in the wild (Dhall, Ramana Murthy, Goecke, Joshi, & Gedeon, 2015).…”

Section: Multimodal Analytics and Professional Activity Detectionmentioning

confidence: 99%

See 1 more Smart Citation

Multimodal teaching analytics: Automated extraction of orchestration graphs from wearable sensor data

Prieto

Sharma

Kidziński

et al. 2018

Computer Assisted Learning

119

121

View full text Add to dashboard Cite

The pedagogical modelling of everyday classroom practice is an interesting kind of evidence, both for educational research and teachers' own professional development. This paper explores the usage of wearable sensors and machine learning techniques to automatically extract orchestration graphs (teaching activities and their social plane over time), on a dataset of 12 classroom sessions enacted by two different teachers in different classroom settings. The dataset included mobile eye-tracking as well as audiovisual and accelerometry data from sensors worn by the teacher. We evaluated both time-independent and time-aware models, achieving median F1 scores of about 0.7-0.8 on leave-one-session-out k-fold cross-validation. Although these results show the feasibility of this approach, they also highlight the need for larger datasets, recorded in a wider variety of classroom settings, to provide automated tagging of classroom practice that can be used in everyday practice across multiple teachers.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Multimodal Analytics and Professional Activity Detectionmentioning

confidence: 99%

Multimodal teaching analytics: Automated extraction of orchestration graphs from wearable sensor data

Prieto

Sharma

Kidziński

et al. 2018

Computer Assisted Learning

119

121

View full text Add to dashboard Cite

show abstract

“…These trends are exemplified in the annual competitions Emotion Recognition in the Wild (EmotiW) [12] and Audio Video Emotion Challenge (AVEC) [13]. Since 2010, deep learning methods have been applied to affect recognition problems across multiple modalities and led to improvements in accuracy, including winning performances at EmotiW [14], [15], [16] and AVEC [17], [18], [19].…”

Section: Introductionmentioning

confidence: 99%

Deep Learning for Human Affect Recognition: Insights and New Developments

Rouast

Adam

Chiong

2021

IEEE Trans. Affective Comput.

160

View full text Add to dashboard Cite

Automatic human affect recognition is a key step towards more natural human-computer interaction. Recent trends include recognition in the wild using a fusion of audiovisual and physiological sensors, a challenging setting for conventional machine learning algorithms. Since 2010, novel deep learning algorithms have been applied increasingly in this field. In this paper, we review the literature on human affect recognition between 2010 and 2017, with a special focus on approaches using deep neural networks. By classifying a total of 950 studies according to their usage of shallow or deep architectures, we are able to show a trend towards deep learning. Reviewing a subset of 233 studies that employ deep neural networks, we comprehensively quantify their applications in this field. We find that deep learning is used for learning of (i) spatial feature representations, (ii) temporal feature representations, and (iii) joint feature representations for multimodal sensor data. Exemplary state-of-the-art architectures illustrate the progress. Our findings show the role deep architectures will play in human affect recognition, and can serve as a reference point for researchers working on related applications.

show abstract

“…Furthermore, the teaching practice categorization used in this study is only one example, and other characterizations are also possible, especially for researchers or practitioners interested in concrete pedagogical approaches such as collaborative learning, or inquiry-based learning (which will also prompt new exploration efforts into different sets of useful multimodal features). Also, further exploration is needed in applying more complex algorithms (e.g., deep/recurrent neural networks), which have recently shown promising capabilities in dealing with rich multimodal data (e.g., [24]). …”

Section: Teacher Activitymentioning

confidence: 99%

Teaching analytics

Prieto

Sharma

Dillenbourg

et al. 2016

Proceedings of the Sixth International Conference on Learning Analytics &Amp; Knowledge - LAK '16

View full text Add to dashboard Cite

ABSTRACT'Teaching analytics' is the application of learning analytics techniques to understand teaching and learning processes, and eventually enable supportive interventions. However, in the case of (often, half-improvised) teaching in face-to-face classrooms, such interventions would require first an understanding of what the teacher actually did, as the starting point for teacher reflection and inquiry. Currently, such teacher enactment characterization requires costly manual coding by researchers. This paper presents a case study exploring the potential of machine learning techniques to automatically extract teaching actions during classroom enactment, from five data sources collected using wearable sensors (eye-tracking, EEG, accelerometer, audio and video). Our results highlight the feasibility of this approach, with high levels of accuracy in determining the social plane of interaction (90%, κ=0.8). The reliable detection of concrete teaching activity (e.g., explanation vs. questioning) accurately still remains challenging (67%, κ=0.56), a fact that will prompt further research on multimodal features and models for teaching activity extraction, as well as the collection of a larger multimodal dataset to improve the accuracy and generalizability of these methods.

show abstract

Hierarchical Committee of Deep CNNs with Exponentially-Weighted Decision Fusion for Static Facial Expression Recognition

Cited by 99 publications

References 28 publications

Multimodal teaching analytics: Automated extraction of orchestration graphs from wearable sensor data

Multimodal teaching analytics: Automated extraction of orchestration graphs from wearable sensor data

Deep Learning for Human Affect Recognition: Insights and New Developments

Teaching analytics

Contact Info

Product

Resources

About