Surface-enhanced Raman spectroscopy (SERS) is a powerful analytical spectroscopy offering advantages ranging from "vibrational fingerprints" to multiplexed detection. However, the use of this technique in real-world applications has been limited due to difficulties in detecting inherently weak Raman signals often embedded in strong interfering background signals. A variety of plasmonics-active platforms have been developed to increase Raman signals but are not sufficient to extract weak SERS signals from intense interfering background signals. Herein, we describe a practical method, referred to as polarization modulation-SERS (PM-SERS), which utilizes the polarization dependence of anisotropic SERS-active nanostructures to modulate the plasmonic effect to extract SERS signals and remove background. The modulation is obtained by switching the polarization of the excitation source at a specific frequency involving addition of only few optical components such as liquid crystal polarizers to a typical Raman setup. In this work, we characterized the polarization-dependent response of the SERS substrates fabricated using the oblique angle evaporation (OAV) technique and their response under laser excitation using a polarization modulated source. We demonstrated that the PM-SERS method can extract the analyte weak SERS signals from the strong interfering background signal in different situations, involving a fluorescent sample and a strong background light, and we show the possibility of using PM-SERS at a quasi-real time rate (0.5 Hz). We believe that the PM-SERS method will help expand the translation of applications that utilize SERS-substrates to realworld settings.