Little is known about the multilevel social determinants of adolescent sexual and reproductive health (SRH) that shape the use of family planning (FP) among young women in Africa. We conducted in-depth, semi-structured, qualitative interviews with 63 women aged 15–24 years in Accra and Kumasi, Ghana. We used purposive, stratified sampling to recruit women from community-based sites. Interviews were conducted in English or local languages, recorded, and transcribed verbatim. Grounded theory-guided thematic analysis identified salient themes. Three primary levels of influence emerged as shaping young women’s SRH experiences, decision-making, and behaviors. Interpersonal influences (peers, partners, and parents) were both supportive and unsupportive influences on sexual debut, contraceptive (non) use, and pregnancy resolution. Community influences included perceived norms about acceptability/unacceptability of adolescent sexual activity and its consequences (pregnancy, childbearing, abortion). Macro-social influences involved religion and abstinence and teachings about premarital sex, lack of comprehensive sex education, and limited access to confidential, quality SRH care. The willingness and ability of young women in our study to use FP methods and services were affected, often negatively, by factors operating within and across each level. These findings have implications for research, programs, and policies to address social determinants of adolescent SRH.