Although previous studies have suggested that hemoglobin is related to the health status of people living with human immunodeficiency virus/acquired immune deficiency syndrome (HIV/AIDS) (PLWHA), the role of anemia in mortality remains unclear. This study aimed to comprehensively quantify the effect of anemia on the mortality risk of PLWHA. In this retrospective cohort study, we thoroughly estimated the effect of anemia on PLWHA mortality, using data collected from January 2005 to June 2022 in the Huzhou area, in 450 subjects extracted from the database of the China Disease Prevention and Control Information System and matched them using a propensity score matching approach to balance potential confounding bias. The potential exposure–response relationship between anemia, hemoglobin concentration, and the mortality of PLWHA was also carefully estimated. A series of subgroup analyses, including interaction analysis, was further conducted to validate the robustness of the effect of anemia on PLWHA death risk. Anemia was significantly associated with an elevated death risk in PLWHA, with an increase of 74% (adjusted hazard ratio [AHR]: 1.74; 95% confidence interval [CI]: 1.03–2.93; p = 0.038) in those with anemia after adjusting for potential confounders. PLWHA with moderate or severe anemia had a higher risk of death, with an 86% increase (AHR = 1.86; 95% CI: 1.01–3.42; p = 0.045). Meanwhile, the AHR tended to increase by 85% on average (AHR = 1.85, 95% CI: 1.37–2.50; p < 0.001) with a per standard deviation (SD) decrease in plasma hemoglobin. Consistent relationships between plasma hemoglobin and the risk of death were further observed in the results from multiple quantile regression models, restricted cubic spline regression models, and a series of subgroup analyses. Anemia is an independent risk factor for HIV/AIDS-related mortality. Our findings may provide new insights into the relevance of PLWHA administration to public health policy, which demonstrate that this low-cost and routinely measured marker (hemoglobin) can be a marker of poor prognosis even before the start of HAART.