Extensive use of pesticides in agricultural production has been causing serious health threats to humans and animals. Among them, phorate is a highly toxic organophosphorus insecticide that has been widely used in planting. Due to its harmful effects on human and animal health, it has been restricted for use in many countries. Analytical methods for the rapid and sensitive detection of phorate residues in agricultural products are urgently needed. In this study, a new method was developed by combining surface-enhanced Raman spectroscopy (SERS) and immunochromatography assay (ICA). Hybrid magnetic Fe3O4@Au@DTNB-Ab nanoprobes were prepared by modifying and growing Au nanoseeds on an Fe3O4 core. SERS activity of the nanoprobe was optimized by adjusting the concentration of the Au precursor. A rapid and sensitive assay was established by replacing the traditional colloidal gold-based ICA with hybrid SERS nanoprobes for SERS-ICA. After optimizing parameters including coating antibody concentrations and the composition and pH of the buffer solution, the limit of detection (LOD) for phorate could reach 1 ng/mL, with a linear range of 5~100 ng/mL. This LOD is remarkably lower than the maximum residue limit in vegetables and fruits set by the Chinese government. The feasibility of this method was further examined by conducting a spiking test with celery as the real sample. The result demonstrated that this method could serve as a promising platform for rapid and sensitive detection of phorate in agricultural products.