Introduction: Men who have sex with men (MSM), and transgender women (TGW), face specific obstacles to retention in care, particularly in settings with stigmatization such as sub-Saharan Africa. We evaluated the impacts of HIV status and other factors on loss-to-follow-up (LTFU) and visit adherence among MSM and TGW in Abuja and Lagos, Nigeria. Methods: TRUST/RV368 is an open cohort that provides comprehensive and integrated prevention and treatment services for HIV and sexually transmitted infections (STIs) at community venues supportive of sexual and gender minorities. Recruitment began in March 2013 and participants were followed every three months for up to 18 months. LTFU was defined as not presenting for an expected visit in the past 180 days. Visit adherence was calculated as a rate of completed visits adjusted by the number of three-month intervals elapsed since enrolment. HIV and other factors predictive of LTFU and visit adherence were evaluated using Cox proportional hazards and Poisson regression models, respectively. Results: A total of 1447 participants who completed enrolment evaluations over two visits as of November 2018 were included in these analyses. Their median age was 24 years (interquartile range [IQR]: 21 to 28) and 53% (n = 766) were living with HIV. LTFU occurred in 56% (n = 808) and visit adherence was 0.62 (95% confidence interval: 0.61 to 0.64) visits per three-month interval. Participants at risk and living with HIV had median follow-up times of 12 months (IQR: 6 to 22), and 21 months (IQR: 12 to 30), respectively (p < 0.01). After controlling for other factors, LTFU was less common among participants living with HIV or other STIs and more common among those who did not own a cell phone, sold sex and had never undergone HIV testing prior to enrolment. These factors had parallel associations with visit adherence. Conclusions: Retention was suboptimal in Nigerian clinics designed to serve MSM and TGW. Particularly high LTFU and low visit adherence among participants at risk for HIV could complicate deployment of HIV prevention interventions. Marketing the benefits of testing, improving access to cell phones and nurturing more trust with clients may improve retention among marginalized communities in Nigeria.