The threat of anthropogenic eutrophication and harmful algal blooms in lakes requires the development of innovative stormwater best management practices (BMPs) to reduce the external loading of phosphorus (P). This paper presents the findings of a 5-year study of a full-scale P removal structure constructed in Minnesota, USA with spent lime drinking water treatment residual (DWTR), a by-product of water softening at a local water treatment plant. Influent and effluent water samples were collected by auto-samplers during 43 storm events during the growing season. Samples were analyzed for P constituents, heavy metals, total suspended solids (TSS), and pH. Toxicity of the effluent was assessed using Ceriodaphnia dubia. Flow-weighted removal effectiveness was calculated for each storm event. Overall, the spent lime DWTR reduced total P loading by 70.9%, dissolved reactive P by 78.5%, dissolved P by 74.7%, and TSS by 58.5%. A significant reduction in heavy metals was also observed. Toxicity tests indicated the aquatic toxicity of the effluent treated with spent lime DWTR was not different from untreated stormwater. This study provided long-term real-world data that demonstrated that a full-scale P removal structure with spent lime DWTR significantly reduced P and other pollutants in stormwater discharging to an urban lake. Therefore, spent lime DWTR, which is currently treated as a waste product, is a promising filter material for stormwater treatment.