People express emotions via a variety of behaviors, including facial muscle movements, body poses and gestures, vocal prosody, and speech. To understand how people experience and perceive emotion, it is crucial to quantify and model these behaviors. However, existing methods are insufficient to address this need. Manually annotating behavior is very time-consuming, making it infeasible to do at scale. Moreover, common linear models cannot fully capture the complex, nonlinear, and interactive affective processes embodied by these behaviors. In this methodology review, we describe how deep learning addresses these challenges and thereby promises to advance naturalistic affective science. First, deep learning provides accessible and efficient tools to annotate dynamic, complex, multi-modal behaviors. These automated annotation tools can scale up behavioral quantification to a degree impossible with human coders, enabling many new, more naturalistic approaches to affective science. Second, deep learning offers innovative paradigms for optimizing and manipulating naturalistic stimuli. This application makes it possible to generate experiment designs with greater generalizability, statistical power, and external validity. Third, deep learning can support flexible, powerful cognitive models of naturalistic affective processing. These novel cognitive models make it possible to explain how the mind and brain engage in the emotional world in ways that are both broader and more precise. However, deep learning is not without its limitations, so we also explore important failure cases, practical issues, and ethical concerns. By detailing the promise and the peril of deep learning, this review paves the way for a more naturalistic and generalizable affective science.
