Facial Expression Modeling and Synthesis for Patient Simulator Systems: Past, Present, and Future

Abstract

Clinical educators have used robotic and virtual patient simulator systems (RPS) for dozens of years, to help clinical learners (CL) gain key skills to help avoid future patient harm. These systems can simulate human physiological traits; however, they have static faces and lack the realistic depiction of facial cues, which limits CL engagement and immersion. In this article, we provide a detailed review of existing systems in use, as well as describe the possibilities for new technologies from the human–robot interaction and intelligent virtual agents communities to push forward the state of the art. We also discuss our own work in this area, including new approaches for facial recognition and synthesis on RPS systems, including the ability to realistically display patient facial cues such as pain and stroke. Finally, we discuss future research directions for the field.

Automatic Facial Palsy Detection—From Mathematical Modeling to Deep Learning

“…Several techniques are proposed in the literature, either based on image analysis, such as combining the left and right sides of the face from two photographs of the same subject performing different facial expressions to generate a new asymmetrical facial expression [51] or simple image transform approaches [82], as well as using generative adversarial networks (GANs) to automatically augment the training data with diverse face images [76,92]. In the same direction, researchers are also starting to focus on building mathematical models of the dynamics of facial expressions for realistic synthetic faces for the modeling of static images and animations [93].…”