Since the discovery of monolayer graphene, the novel physical properties of two-dimensional (2D) materials have garnered significant attention, particularly for those with fewer layers that often exhibit unique properties distinct from bulk materials. Therefore, accurately determining the layer count or obtaining the microscopic surface morphology is crucial in both the laboratory and during device manufacturing. However, traditional detection methods have numerous drawbacks. There is an urgent need for a convenient, accurate, and non-destructive scientific method to characterize the layer number and surface microstructure of 2D materials. Combined with the experimental setup of laser scanning photocurrent spectroscopy, we have developed a polarization-modulated scanning optical microscope based on the principle of Reflectance Difference Spectroscopy (RDS). By monitoring the reflectivity of the samples, we can observe changes in the reflection signal strength of MoS<sub>2</sub> with different layer numbers. The intensity of the reflectance differential spectral signal reflects changes in the layer count within the sample. We can characterize the changes in the number of layers of two-dimensional materials in a non-contact manner using polarization-modulated scanning optical microscopy. Through the study of the reflectance differential spectra of two typical 2D layered materials, MoS<sub>2</sub> and ReSe<sub>2</sub>, we found that our polarization-modulated scanning optical microscope system is also more sensitive to the characteristics of the stacking anisotropy of the 2D materials than the conventional reflection microscope. This indicates that our research contributes to a better understanding of the layer number characteristics and anisotropic properties of layered 2D materials. What’s more, our research also provides a non-contact optical method to characterize the number of layers and optical anisotropy of two-dimensional layered materials.