Emotions Matter: Towards Personalizing Human-System Interactions Using a Two-layer Multimodal Approach

Kalatzis, Apostolos; Prabhu, Vishnunarayan Girishan; Rahman, Saidur; Wittie, Mike P.; Stanley, Laura

doi:10.1145/3536221.3556582

Cited by 2 publications

(2 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Figure 6 gives changes in the affection classification accuracy corresponding to different key area numbers, showing the relationship between key area number and affection classification accuracy. For the given six key area numbers (1,2,3,4,5,6), the corresponding affection classification accuracy is 77%, 77.5%, 81.9%, 82.7%, 83.1%, and 83.4%, respectively. Analysis of table data shows that with the increase of key area number, the affection classification accuracy shows an upward trend, indicating more key areas can help the model better capture affection information and improve affection classification accuracy.…”

Section: Experimental Results and Analysismentioning

confidence: 99%

“…As AI and deep learning are both developing fast these days, significant progresses have been made in fields of computer vision and natural language processing [1][2][3][4][5], which has created a background for the emergence of MAC for combined processing of texts and images [6][7][8][9][10][11][12]. Conventional teaching evaluation emphasizes students' knowledge mastery over their affections [13][14][15], however, the state of affection has a very important influence on students' learning process and outcome.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Modal Affective Computing: An Application in Teaching Evaluation Based on Combined Processing of Texts and Images

Wang¹,

Zhai²,

Xu³

et al. 2023

View full text Add to dashboard Cite

Conventional teaching evaluation emphasizes students' knowledge mastery over their affections. Multi-modal Affective Computing (MAC) can analyze versatile information of students in the classroom, including their facial expressions, gestures, and text feedback, in a comprehensive way, thereby helping teachers discover problems with students' affections in a timely manner, so that they could adjust the teaching methods and strategies accordingly. However, the available MAC technology might make unstable or wrong judgement when dealing with complex affective expressions, then the inaccurate evaluation results of students' affection state might adversely affect the teaching evaluation results. To tackle these issues, this study innovatively applied MAC in teaching evaluation based on combined processing of texts and images. The input texts were divided into two parts: main body and the hash tag, which were subjected to feature extraction respectively. The image features were extracted from two angles: object and scene, since the two angles can give image information of different levels. The MAC model was divided into modal sharing tasks and modal private tasks to attain better adaptability in case of new teaching evaluation scenarios. The effectiveness of the proposed method was verified by experimental results.

show abstract