The incidence of disease arising from food-borne pathogens is increasing continuously and has become a global public health problem. Rapid and accurate identification of food-borne pathogens is essential for adopting disease intervention strategies and controlling the spread of epidemics. Surface-enhanced Raman spectroscopy (SERS) has attracted increasing interest due to the attractive features including simplicity, rapid measurement, and high sensitivity. It can be used for rapid in situ sensing of single and multicomponent samples within the nanostructure-based confined space by providing molecular fingerprint information and has been demonstrated to be an effective detection strategy for pathogens. This article aims to review the application of SERS to the rapid sensing of food-borne pathogens in food matrices. The mechanisms and advantages of SERS, and detection strategies are briefly discussed. The latest progress on the use of SERS for rapid detection of food-borne bacteria and viruses is considered, including both the labeled and label-free detection strategies. In closing, according to the current situation regarding detection of food-borne pathogens, the review highlights the challenges faced by SERS and the prospects for new applications in food safety.