We know that HIV treatment outcome depends on antiretroviral treatment (ART) adherence. Young adults with perinatal HIV (YPHIV) who survived have endured various adherence challenges in their adolescent years. While some of them could maintain perfect adherence with sustainable virologic suppression, many experienced one or more episodes of virologic failure. We explored factors affecting ART adherence from real-life experiences of YPHIV. A qualitative study was conducted between June and November 2022. Twenty YPHIV aged 21–29 years with a history of virologic failure and resumed virologic suppression during adolescent years were invited to share their experiences through individual in-depth interviews. Audio records were transcribed verbatim and analyzed using deductive thematic analysis. We divided excerpts into two themes: barriers and facilitators to ART adherence. The socio-ecological model was used to frame subthemes at personal, societal, and healthcare system levels. Most barriers to adherence were concentrated at the personal level, including work/study-related conditions, personal entertainment, medication issues, mental health problems, thought, and belief. At the societal level, social activities and fear of HIV disclosure were frequently mentioned as barriers. Medical care cost was the only identified barrier at the healthcare system level. The facilitators to adherence at the personal level included perceiving health deterioration, being afraid of hospitalization and medical procedures, and wishing to be healthy and move on. At the same time, perceived family support and determination to complete family without HIV transmission were identified as facilitators at the societal level. Service behaviors of healthcare providers were mentioned as facilitators to adherence at the healthcare system level. From this study, most factors associated with non-adherence in adolescents were at the personal level, and the fear of HIV disclosure was critical at the societal level. The key facilitator to adherence was the determination to be healthy and have a promising future. Our findings reinforce the importance of establishing youth-friendly services in the existing HIV care setting. More time allocation for tailored individual counseling, using other novel approaches like mHealth, online media, and involvement of social support from different sectors might be beneficial to maximize adherence self-efficacy during the transitional period of YPHIV.