Permeable pavement is a low impact development technology for stormwater (SW) runoff control and pollutant removal. The strength of SW depends on land use of the catchment, e.g., semi-urban vs. industrial. The performance (in terms of pollutants removal) of permeable clay bricks (PCB) has not been adequately assessed for SW of varying strengths. For using the permeable clay bricks as a pavement surface layer, the present research investigates its pollutant removal capacity through SW infiltration using a laboratory setup. SW samples of two different strengths, i.e., high polluted stormwater (HPSW) and less polluted stormwater (LPSW), were tested for a pavement system consisting of the clay brick layer on top of a coarse gravel support layer. The tests were performed at a rainfall intensity of 12.5 mm/h (for a 10-year return period in Buraidah, Qassim) to evaluate the suitability of PCB for the arid and semi-arid regions. The experiments revealed that PCB became fully saturated and achieved a steady-state outflow condition after 10 min of rainfall. Irrespective of contamination level, the pollutant removal efficiency was found to be similar for both HPSW and LPSW. High TSS (>98%) and turbidity (>99%) removals were achieved for both strengths, while BOD5 (78.4%) and COD (76.1%) removals were moderate. Poor to moderate nutrient removal, 30.5% and 39.1% for total nitrogen (TN) and 34.7% and 31.3% for total phosphorus (TP), respectively for HPSW and LPSW, indicates an adsorptive removal of nutrients in the system. Heavy metal removal efficiency ranged from 6.7% to 94%, with higher removals archived for Fe, Mn, Se, and Pb. The study provides insights into the role of PCB as a surface layer in the permeable pavement for pollutant removal. The study also establishes the guidelines for the optimal permeable pavement design to deal with SW of varying contamination levels. Permeable clay bricks showed the potential to be used as a sustainable LID technology for arid regions.