Agricultural pollutants are harmful components threatening human health, wildlife, the environment, and the ecosystem. To avoid their exposure, developing prevention and detection systems with high sensitivity and selectivity is required. Most conventional methods, including molecular and chromatographic techniques, cannot be adopted for outdoor on-site detection even though they can provide sensitive and selective detection. Thus, detection platforms that can provide on-site detection via miniaturized and high throughput systems should be developed. As an alternative method, surface-enhanced Raman scattering (SERS) provides unique information about the substances in the presence of plasmonic nanostructures, and it can be portable with the use of portable detection systems and spectrometers. In this study, on-site detection of agricultural pollutants through SERS is reviewed. Three different types of agricultural pollutants were pointed out. On-site detection of biological pollutants, including bacteria and viruses, is reviewed as the first type of pollutant. As a second type, the detection of pesticides, antibiotics, and additives are focused on as chemical pollutants. The third group includes the detection of microplastics and also nanoparticles from the environment.