Some people are exceptional at reading emotional expressions, while others struggle. Here we ask whether the way we experience emotion “on the inside” influences the way we expect emotions to be expressed in the “outside world” and subsequently our ability to read others’ emotional expressions. Across multiple experiments, incorporating discovery and replication samples, we develop EmoMap (N = 20; N = 271) and ExpressionMap (N = 98; replication N = 193) to map adults’ experiences of emotions and visual representations of others’ emotions. Some individuals have modular maps, wherein emotional experiences and visual representations are consistent and distinct—anger looks and feels different from happiness, which looks and feels different from sadness. In contrast, others have experiences and representations that are variable and overlapping—anger, happiness, and sadness look and feel similar and are easily confused for one another. Here we illustrate an association between these maps: those with consistent and distinct experiences of emotion also have consistent and distinct visual representations of emotion. Finally (N = 193), we construct the Inside Out Model of Emotion Recognition, which explains 60.8% of the variance in emotion recognition and illuminates multiple pathways to emotion recognition difficulties. These findings have important implications for understanding the emotion recognition difficulties documented in numerous clinical populations.