The efficient capture of multi-pollutant residues in food is vital for food safety monitoring. In this study, in-situ-fabricated magnetic MIL-53(Al) metal organic frameworks (MOFs), with good magnetic responsiveness, were synthesized and applied for the magnetic solid-phase extraction (MSPE) of chloramphenicol, bisphenol A, estradiol, and diethylstilbestrol. Terephthalic acid (H2BDC) organic ligands were pre-coupled on the surface of amino-Fe3O4 composites (H2BDC@Fe3O4). Fe3O4@MIL-53(Al) MOF was fabricated by in-situ hydrothermal polymerization of H2BDC, Al (NO3)3, and H2BDC@Fe3O4. This approach highly increased the stability of the material. The magnetic Fe3O4@MIL-53(Al) MOF-based MSPE was combined with high-performance liquid chromatography-photo diode array detection, to establish a novel sensitive method for analyzing multi-pollutant residues in milk. This method showed good linear correlations, in the range of 0.05–5.00 μg/mL, with good reproducibility. The limit of detection was 0.004–0.108 μg/mL. The presented method was verified using a milk sample, spiked with four pollutants, which enabled high-throughput detection and the accuracies of 88.17–107.58% confirmed its applicability, in real sample analysis.