Metal isotope abundance detection is significant in earth science, environmental monitoring and industrial applications. A new technique for the detection of metal isotopes was developed by wavelength modulation spectroscopy technology in this work. The measurements were made for the rubidium transition at 795 nm using four kinds of rubidium compound samples (RbCl, Rb2CO3, Rb2SO4, and RbI). Based on the redox reaction, the solid rubidium compound was mixed with the reducing agent. The micro-channel array structure of the atomic generator can produce highly collimated atomic vapor. The ability of high-order harmonics (2nd, 4th, and 6th) to suppress system noise was compared, and the result showed that the fourth-harmonic signal not only enhanced signal-to-noise ratio but also ensured effective spectral resolution. Compared with direct absorption, the signal-to-noise ratio of the 4th harmonic was enhanced by approximately 3.3 times. In addition, wavelet denoising further suppressed the baseline standard deviation of the system by about a factor of 1.