The identification of the illicit substances in low‐purity seized drugs with Raman spectroscopy is an outstanding problem. The low concentration of the target molecule demands high sensitivity and the presence of impurities produces a strong fluorescence background that makes identification challenging. Although Raman analyzers with 785 and 830 nm excitation wavelengths provide high sensitivity, the longer 1064 nm wavelength produces lower fluorescence. In this work, we demonstrate a Raman spectrometer that simultaneously achieves both high sensitivity and low background fluorescence. We utilize swept‐source Raman spectroscopy in which the spectrometer is replaced by a single high‐collection spectral channel, and the Raman spectrum is swept using a tunable laser. By eliminating the spectrometer and its slit, the optical throughput and sensitivity are improved. Moreover, our swept‐source system requires a single uncooled silicon photodiode that provides higher quantum efficiency at longer wavelengths compared with CCDs. This allowed us to use excitation wavelengths in the 900 nm range to reduce background fluorescence without sacrificing sensitivity. We have demonstrated 6x background fluorescence reduction in colored seized drugs compared with 830 nm excitation. With lower background fluorescence and high sensitivity, we could identify heroin with only 1% purity in a seized sample. Besides forensics and security, the demonstrated swept‐source Raman spectroscopy approach can be a powerful technique in biomedical applications where strong background fluorescence limits detection capabilities.