Objectives Pill count is used to assess drug adherence in people living with HIV (PLHIV). Carrying a pillbox is associated with fear of concealment and stigma and might indicate poor adherence and predict someone who will be lost to follow‐up (LTFU). We therefore assessed the association between pillbox return and being LTFU in rural Tanzania. Methods This is a nested study of the Kilombero and Ulanga Antiretroviral Cohort (KIULARCO). We included PLHIV aged ≥ 18 years enrolled in KIULARCO between January 2013 and March 2019 with follow‐up through January 2020, who were on antiretroviral treatment (ART) for ≥ 6 months. Baseline was defined as the latest ART initiation or KIULARCO enrolment. We determined the association between time‐dependent failed pillbox return updated at every visit and LTFU using Kaplan–Meier estimation and Cox models. Results Among 2552 PLHIV included in the study, 1735 (68.0%) were female, 959 (40.3%) had a WHO stage III/IV and 1487 (66.4%) had a CD4 cell count < 350 cells/µL. The median age was 38.4 years [interquartile range (IQR): 31.7–46.2]. During a median follow‐up of 33.1 months (IQR: 17.5–52.4), 909 (35.6%) participants were LTFU, 43 (1.7%) died and 194 (7.6%) had transferred to another clinic. The probability of being LTFU was higher among PLHIV with failed pillbox return than among those who returned their pillbox [30.0%, 95% confidence interval (CI): 26.8–33.2% vs. 19.4%, 95% CI: 17.4–21.6%, respectively, at 24 months (hazard ratio = 1.67, 95% CI: 1.46–1.90; p < 0.001)]. Conclusions Failed pillbox return was associated with a higher risk of being LTFU and could be used as a simple tool to identify PLHIV for appropriate interventions to reduce their chance of being LTFU.