We conducted a series of activation likelihood estimation (ALE) meta-analyses to determine the commonalities and distinctions between separate levels of emotion perception, namely incidental perception, passive perception, and explicit evaluation of emotional expressions. Pooling together more than 180 neuroimaging experiments using facial, vocal or body expressions, our results are threefold. First, explicitly evaluating the emotions of others recruits brain regions associated with the sensory processing of expressions, such as the inferior occipital gyrus, middle fusiform gyrus and the superior temporal gyrus, and brain regions involved in low-level and high-level mindreading, namely the posterior superior temporal sulcus, the inferior frontal cortex and dorsomedial frontal cortex. Second, we show that only the sensory regions were also consistently active during the passive perception of emotional expressions. Third, we show that the brain regions involved in mindreading were active during the explicit evaluation of both facial and vocal expressions. We discuss these results in light of the existing literature and conclude by proposing a cognitive model for perceiving and evaluating the emotions of others.