The main objective of this paper is to understand the role of the family on the sexual experiences of adolescents from urban, disadvantaged settings in Baltimore and Johannesburg. Data were collected as part of the WAVE study, a global study of disadvantaged youth in five cities. Qualitative data were based on key informant interviews, a Photovoice exercise, community mapping, focus groups and in-depth interviews with adolescents. Quantitative data were gathered from an ACASI survey that was administered to approximately 450–500 adolescents per site. Results from the qualitative data revealed that while parents were viewed as important sources of information for sexual and reproductive health, they were often not present in the adolescents’ lives. This lack of parental presence was perceived to result in adolescents feeling an overall lack of adult support and guidance. The impact of parental presence and support on adolescent sexual experience was further examined from the quantitative data and revealed a complex picture. In both Baltimore and Johannesburg, female adolescents who were raised by other relatives were less likely to report having had sex compared to those raised by two biological parents, which was not observed for males. In Johannesburg, female adolescents who were paternal orphans were less likely to have had sex compared to non-orphans; the opposite was true among males. Finally, in both sites, female adolescents who had been exposed to violence were more likely to have had sex compared to those who had not; for males, there was no significant relationship. The study demonstrates the powerful influence of both context and gender for understanding the influences of the family on adolescent sexual behaviors. Programs aiming to reduce adolescent sexual risk behaviors the need to understand the complex influences on risk behaviors in different settings and in particular, the role of mothers and fathers. Prevention strategies need to also understand and incorporate gender-specific messages and interventions in order to address the high risk of sexual behaviors among adolescents in these settings.