After more than two decades of the expansion of antiretroviral treatment (ART) in adolescents living with perinatal HIV (APHIV) in sub-Saharan Africa, there is still poorly sustained long-term adherence to ART due to multifactorial factors with the consequence of increased mortality and morbidity. There are little data available on the familial and structural factors which affect sustenance to long-term adherence to ART. A qualitative exploratory design was used to conduct in-depth interviews with 21 APHIV attending HIV care and management in the rural health facilities of Vhembe district in Limpopo Province, South Africa. Transcripts were translated verbatim into English, and data were analyzed using Tesch’s eight steps of qualitative data analysis. The sample consisted of APHIV 10–19 years old who were aware of their HIV status, and all had received ART for more than 5 years. They lived in extended, disrupted, grandparent- and child-headed households. They experienced food insecurities due to poverty or orphanhood, as well as living in disrupted households, which deterred them from long-term adherence. In addition, dependency on social support grants to sustain their livelihoods affected long-term adherence. APHIV had challenges with structural factors such as inconsistent clinic attendance, clashes between school activities and clinic appointments, and the lack of transport fare to the clinic, which affected adherence. Although APHIV were on one-pill fixed-dose ART, they were not able to sustain long-term adherence due to various familial, structural, and psychosocial challenges. In addition to institution-based interventions, there is a need for family, community-based, and multi-sectorial interventions to support long-term ART adherence among APHIV.