The worldwide mining industry produces millions of tons of rock wastes, raising a considerable burden for managing both economic and environmental issues. The possible reuse of Fe/Mn-rich materials for arsenic removal in water filtration units, along with rock properties, was evaluated. By characterizing and testing 47 samples collected from the Joda West Iron and Manganese Mine in India, we found As removal up to 50.1% at 1 mg/L initial As concentration, with a corresponding adsorption capacity of 0.01–0.46 mgAs/g mining waste. The As removal potential was strictly related to spectral, mineralogical, and elemental composition of rock wastes. Unlike rock crystallinity due to quartz and muscovite, the presence of hematite, goethite, and kaolinite, in association with the amorphous fractions of Fe and Al, enhanced the As adsorption. The natural content of arsenic indicated itself the presence of active sorptive sites. The co-occurrence of site-specific competitors (i.e., phosphate) represented a consequent limitation, whereas the content of Ce, Cu, La, and Pb contributed positively to the As adsorption. Finally, we proposed a simplified multiple linear model as predictive tool to select promising rock wastes suitable for As removal by water filtration in similar mining environments: As predicted = 0.241 + 0.00929[As] + 0.000424[La] + 0.000139[Pb] − 0.00022[P].