Innovative application of surface-enhanced Raman scattering (SERS) for rapid and nondestructive analyses has been gaining increasing attention for food safety and quality. SERS is based on inelastic scattering enhancement from molecules located near nanostructured metallic surfaces and has many advantages, including ultrasensitive detection and simple protocols. Current SERS-based quality analysis contains composition and structural information that can be used to establish an electronic file of the food samples for subsequent reference and traceability. SERS is a promising technique for the detection of chemical, biological, and harmful metal contaminants, as well as for food poisoning, and allergen identification using label-free or label-based methods, based on metals and semiconductors as substrates. Recognition elements, including immunosensors, aptasensors, or molecularly imprinted polymers, can be linked to SERS tags to specifically identify targeted contaminants and perform authenticity analysis. Herein, we highlight recent studies on SERS-based quality and safety analysis for different foods categories spanning the whole food chain, ‘from farm to table’ and processing, genetically modified food, and novel foods. Moreover, SERS detection is a potential tool that ensures food safety in an easy, rapid, reliable, and nondestructive manner during the COVID-19 pandemic.