This study aimed to determine the relationship between hemoglobin (Hb) concentration and post-operative delirium (POD) in elderly patients undergoing femoral neck fracture (FNF) surgery and to investigate whether the change in Hb concentration is associated with POD and the risk factors for POD. A total of 889 patients admitted with FNF between January 2016 and December 2020 were enrolled in this single-center, retrospective, case–control study. Hb concentrations were determined at admission and post-operative day 1 and the change in Hb concentration was defined as the absolute value of difference in pre-operative and post-operative Hb concentration. POD was assessed using the Confusion Assessment Method for the Intensive Care Unit (ICU) or the Confusion Assessment Method once a daily after surgery. The logistic regression analysis was performed for statistical analysis. In total, 172 (19.3%) patients developed POD and 151 (87.8%) patients developed POD within post-operative 3 days. Low pre-operative Hb concentration [p = 0.026, odds ratio (OR) = 0.978] and significant change in Hb concentration (p = 0.006, OR = 1.033) were significantly associated with POD. After excluding change in Hb concentration or pre-operative Hb concentration, neither of them was significantly associated with POD (p > 0.05). The interaction analysis of change in Hb concentration and pre-operative Hb concentration in the logistic regression model was negative. There was no significant relationship between post-operative Hb concentration and POD. Age (p < 0.001, OR = 1.072), stroke history (p = 0.003, OR = 2.489), post-operative ICU transfer (p = 0.007, OR = 1.981), and visual analog scale score within post-operative 2 days (p1 = 0.016 and p2 = 0.006) were independently associated with POD in the logistic regression analysis. Patients with low pre-operative Hb concentrations and high changes in Hb concentration seem to have an increased risk of POD and should receive more attention. Old age, stroke history, post-operative ICU transfer, and pain within post-operative 2 days were significantly associated with POD.