Background: High participant retention is essential to achieve adequate statistical power for clinical trials. We assessed participant retention and predictors of loss to follow-up (LTFU) in an HIV vaccine-preparedness study in Masaka, Uganda. Methods: Between July 2018 and March 2021, HIV sero-negative adults (18–45 years) at high risk of HIV infection were identified through HIV counselling and testing (HCT) from sex-work hotspots along the trans-African highway and fishing communities along the shores of Lake Victoria. Study procedures included collection of baseline socio-demographic data, quarterly HCT, and 6-monthly collection of sexual risk behaviour data. Retention strategies included collection of detailed locator data, short clinic visits (1–2 h), flexible reimbursement for transport costs, immediate (≤7 days) follow-up of missed visits via phone and/or home visits, and community engagement meetings. LTFU was defined as missing ≥2 sequential study visits. Poisson regression models were used to identify baseline factors associated with LTFU. Results: 672 participants were included in this analysis. Of these, 336 (50%) were female and 390 (58%) were ≤24 years. The median follow-up time was 11 months (range: 0–31 months). A total 214 (32%) participants were LTFU over 607.8 person-years of observation (PYO), a rate of 35.2/100 PYO. LTFU was higher in younger participants (18–24 years versus 35–45 years, adjusted rate ratio (aRR) = 1.29, 95% confidence interval (CI) 0.80–2.11), although this difference was not significant. Female sex (aRR = 2.07, 95% CI, 1.51–2.84), and recreational drug use (aRR = 1.61, 95% CI, 1.12–2.34) were significantly associated with increased LTFU. Engagement in transactional sex was associated with increased LTFU (aRR = 1.36, 95% CI, 0.97–1.90) but this difference was not significant. LTFU was higher in 2020–2021 (the period of COVID-19 restrictions) compared to 2018–2019 (aRR = 1.54, 1.17–2.03). Being Muslim or other (aRR = 0.68, 95% CI 0.47–0.97) and self-identification as a sex worker (aRR = 0.47, 95% CI, 0.31–0.72) were associated with reduced LTFU. Conclusion: We observed a high LTFU rate in this cohort. LTFU was highest among women, younger persons, recreational drug users, and persons who engage in transactional sex. Efforts to design retention strategies should focus on these subpopulations.