Surface imperfections or contamination on an optical smooth surface is usually inevitable and causes scattering. The directional information of the scattered ray can be related to the spatial frequency through the grating equation. On the other hand, the layout of the optical system determines whether a scattered ray will finally reach the detector according to the positions and directions of the scattered rays. Therefore, the scattering propagation in an optical system is usually band-limited and the effective optical range differs for different systems. In this paper, a method based on ray tracing is described that can statistically determine the optical scattering and the band of roughness. The results can be an essential reference for optical surface polishing and contamination control.