The stormwater management practices have changed from conveyance-oriented to storage-oriented, as part of the Best Management Practices (BMPs). Still, uncontrolled development increases potential pollutants in the stormwater, which conveys into a water body or river. Continuous improvements in the filtration mechanism would complement stormwater management. For the past decades, there is progress in applying granular filter media for stormwater quality improvement. However, the reports were not systematically reviewed. In this paper, the recent five years research that utilizes granular filter media for improving stormwater quality was retrieved and reviewed. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) was referred to, where Scopus and Web of Sciences, two primary journal databases, were used. Initially, keywords searching strings have resulted in 467 articles, which were further screened. Four themes have been formed: stormwater management, stormwater characteristics, separation mechanisms, and future perspectives. Next, two sub-themes and two sub-sub-themes were further established. Then, 65 articles were included manually to complement the themes developed to explore the potential agro-industrial wastes as sustainable filter media. Therefore, this review has proven that the relatively inexpensive and renewable resources from the agro-industrial wastes can remove pollutants efficiently from the stormwater. Four main criteria affecting filter media performances are also highlighted, including the grain sizes of the media, media bed configuration, hydraulic loading rate, and the suspended solids concentration. Further study on these variables can be beneficial to explore the impact of utilizing agro-based media in stormwater filtration.