Ecological floating beds, with their compact footprint and mobility, offer a promising solution for sustainable surface water remediation in rural areas. However, low removal efficiency and instability still limit its application. In this study, iron–carbon-based fillers were integrated into ecological floating beds to investigate their impact and mechanisms in removing pollutants, including carbon, nitrogen, phosphorus, and heavy metals. Results indicate that all five fillers (activated carbon, iron–carbon fillers, sponge iron, activated carbon + iron–carbon fillers, and activated carbon + sponge iron) can completely remove orthophosphate, and the sponge iron filler system can completely remove nitrate. Then, fillers were applied to ecological floating beds, and the iron–carbon microelectrolysis (activated carbon + sponge iron filler)-enhanced ecological floating bed showed superior removal efficiency for pollutants. It achieved 95% removal of NH4+-N, 85% removal of NO3−-N, 75% removal of total phosphorus, 90% removal of chemical oxygen demand, and 90% removal of heavy metals. Typical nitrifying bacteria Nitrospira, denitrifying bacteria Denitratisoma, and a variety of bacterial genera with denitrification functions (e.g., Rhodobacter, Dechloromonas, Sediminibacterium, and Novosphingobium) coexisted in the system, ensuring efficient and robust nitrogen removal performance. These findings will provide support for the sustainable treatment of surface water in rural areas.