While most parents and health care providers understand the importance of educating young people about their emerging sexuality, many report never discussing sex with the young people in their care. Using data from a survey of 1,193 emerging adults, we applied concept mapping to a corpus of over 2,350 short qualitative responses to two questions: (a) What, if anything, makes it difficult to talk to your parents about sexuality or your sexual health? and (b) What, if anything, makes it difficult to talk to your doctors, therapists, or mental health professionals about sexuality or your sexual health? Qualitative analyses revealed that while embarrassment, shame, and awkwardness were commonly reported barriers to communicating with both parents and providers, participants reported different effects across settings: Parent-related embarrassment was associated with concerns about changing the intimacy of the parental relationship, while provider-related embarrassment was associated with fears of seeming incompetent or eliciting dismissal. These observations were supported by multidimensional scaling and hierarchical cluster analyses, which we used to derive conceptual maps based on quantitative spatial analysis of single-concept statements. These analyses revealed a best-fit solution of eight conceptual groups for barriers to discussing sexuality with health care providers, but only four groups of barriers in discussing with parents. Broadly, our findings reinforce the need to tailor sexual health communication to patient characteristics and settings.