Adaptive user interface design and analysis using emotion recognition through facial expressions and body posture from an RGB-D sensor

Medjden, Selma; Ahmed, Naveed; Lataifeh, Mohammed

doi:10.1371/journal.pone.0235908

Cited by 14 publications

(4 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent research has explored the applications of emotion recognition and eye-tracking in web interaction design. These studies have demonstrated how the real-time detection of user emotions can lead to adaptive interfaces that tailor content based on emotional feedback [7]. Additionally, eyetracking data has been used to identify areas of interest on web pages, allowing designers to optimize the placement of critical elements [8].…”

Section: Literature Reviewmentioning

confidence: 99%

Enhancing User Experience Using a Framework Integrating Emotion Recognition and Eye-Tracking

Wang

2024

HSET

View full text Add to dashboard Cite

In a rapidly evolving digital landscape, ensuring a seamless and engaging user experience (UX) has become paramount. This study delves into the intricate realm of web interaction design, aiming to enhance user satisfaction and engagement. Through the integration of emotion recognition and eye-tracking technologies, a profound relationship between user emotions and web interface design is unveiled. The central theme of this research revolves around the integration of emotion recognition and eye-tracking technologies to evaluate web interaction designs. An experimental study involving 24 participants from diverse backgrounds and age groups was conducted. These participants navigated web interfaces that featured both emotion recognition and eye-tracking technologies. Three distinct tasks were carefully crafted to represent a spectrum of web interactions, ranging from basic navigation to complex decision-making. Data collection involved real-time valence and arousal scores, video recordings, visual attention patterns, task completion times, and questionnaires. The analysis revealed a series of significant discoveries. Users who engaged with simplified web versions consistently exhibited elevated valence values, indicative of heightened positive emotional feedback, diverging starkly from their counterparts navigating complex web iterations. The dynamic ebb and flow of emotional states, underscored by arousal levels, underscore the pivotal role of real-time emotional assessment in the critical evaluation of web interfaces. Moreover, the study unveiled a persistent preference for a state of emotional calmness during interactions, demonstrating a universal need for user-centered design principles that prioritize minimal cognitive load and emotional tranquility. In summation, this research contributes a robust framework to the design community and academia for the comprehensive evaluation of web interaction designs. The findings underscore the paramount significance of simplicity, real-time emotional evaluation, and unwavering adherence to user-centered design principles in the realm of web interaction. This study constitutes an invaluable repository for designers, developers, and researchers, steering their endeavors towards the relentless pursuit of optimized user experiences within the ever-evolving digital landscape.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Enhancing User Experience Using a Framework Integrating Emotion Recognition and Eye-Tracking

Wang

2024

HSET

View full text Add to dashboard Cite

show abstract

“…This summary shown in Table I, encapsulates the cross-validation method used, the resolution to which images were resized, and the normalization parameters for RGB channels [10] applied to the AffectNet dataset. These characteristics are crucial for understanding the dataset's structure and preparing it for effective model training and evaluation in computer vision tasks.…”

Section: Data Sectionmentioning

confidence: 99%

Improving Facial Expression Classification through Ensemble Deep Learning Models

2024

nano-ntp

View full text Add to dashboard Cite

This article presents a comprehensive study on improving facial expression classification through the deployment of an ensemble deep learning model that amalgamates multiple advanced CNN architectures, including Inception V3 (Inception Architecture Version-3), ResNet50 (Residual Network with 50 layers), and SPP-net (Spatial Pyramid Pooling in Deep Convolutional Networks). By leveraging the strengths of these diverse models, the ensemble approach aims to capture a richer representation of facial features, thereby enhancing classification accuracy. The study meticulously outlines the data preprocessing techniques employed, such as image segmentation and normalization, to prepare the AffectNet dataset for model training and evaluation. On the AffectNet dataset, the proposed ensemble model achieves 85.2% classification accuracy, outperforming the best individual CNN by 5-10%. This marks a significant improvement, showcasing the potential of ensemble methods in the field of facial expression recognition. The article highlights future research directions, including the exploration of end-toend trainable ensemble models and the collection of more diverse datasets to further refine and enhance model performance. This study contributes to the ongoing advancements in affective computing by demonstrating the effectiveness of ensemble deep-learning models in improving facial expression classification accuracy. It opens new avenues for research and application, promising to enrich human-computer interactions and deepen our understanding of human emotions.

show abstract

“…The application of emotion recognition methods in UX evaluation, however, has not always been so straightforward. As UX has become an essential process to measure the user's satisfaction and usability, for which emotion can act as a key aspect for evaluating practical applications or software products [3].…”

Section: Introductionmentioning

confidence: 99%

A Hybrid Multimodal Emotion Recognition Framework for UX Evaluation Using Generalized Mixture Functions

Razzaq

Hussain

Bang

et al. 2023

Sensors

View full text Add to dashboard Cite

Multimodal emotion recognition has gained much traction in the field of affective computing, human–computer interaction (HCI), artificial intelligence (AI), and user experience (UX). There is growing demand to automate analysis of user emotion towards HCI, AI, and UX evaluation applications for providing affective services. Emotions are increasingly being used, obtained through the videos, audio, text or physiological signals. This has led to process emotions from multiple modalities, usually combined through ensemble-based systems with static weights. Due to numerous limitations like missing modality data, inter-class variations, and intra-class similarities, an effective weighting scheme is thus required to improve the aforementioned discrimination between modalities. This article takes into account the importance of difference between multiple modalities and assigns dynamic weights to them by adapting a more efficient combination process with the application of generalized mixture (GM) functions. Therefore, we present a hybrid multimodal emotion recognition (H-MMER) framework using multi-view learning approach for unimodal emotion recognition and introducing multimodal feature fusion level, and decision level fusion using GM functions. In an experimental study, we evaluated the ability of our proposed framework to model a set of four different emotional states (Happiness, Neutral, Sadness, and Anger) and found that most of them can be modeled well with significantly high accuracy using GM functions. The experiment shows that the proposed framework can model emotional states with an average accuracy of 98.19% and indicates significant gain in terms of performance in contrast to traditional approaches. The overall evaluation results indicate that we can identify emotional states with high accuracy and increase the robustness of an emotion classification system required for UX measurement.

show abstract

Adaptive user interface design and analysis using emotion recognition through facial expressions and body posture from an RGB-D sensor

Cited by 14 publications

References 37 publications

Enhancing User Experience Using a Framework Integrating Emotion Recognition and Eye-Tracking

Enhancing User Experience Using a Framework Integrating Emotion Recognition and Eye-Tracking

Improving Facial Expression Classification through Ensemble Deep Learning Models

A Hybrid Multimodal Emotion Recognition Framework for UX Evaluation Using Generalized Mixture Functions

Contact Info

Product

Resources

About