Surface imperfections or contamination on an optical smooth surface is usually inevitable and causes scattering. The directional information of the scattered ray can be related to the spatial frequency through the grating equation. On the other hand, the layout of the optical system determines whether a scattered ray will finally reach the detector according to the positions and directions of the scattered rays. Therefore, the scattering propagation in an optical system is usually band-limited and the effective optical range differs for different systems. In this paper, a method based on ray tracing is described that can statistically determine the optical scattering and the band of roughness. The results can be an essential reference for optical surface polishing and contamination control.

Space-based gravitational wave detection programs, such as the Laser Interferometer Space Antenna (LISA) or Taiji program, obtain gravitational wave signals by measuring the change in the distance between three satellites by laser. The telescope is an important part of the measurement system, and its function is to transmit and receive laser signals. Due to changes in the space environment, the telescope will inevitably introduce additional dynamic aberrations, which will bring optical path errors to the inversion of gravitational wave signals. Taking LISA as an example, to achieve pm-level measurement accuracy at the detection frequency of 0.1 mHz–1 Hz, the stability requirements of the telescope are less than 1 pm/Hz1/2. This paper theoretically deduces the aberration types that affect the telescope’s stability and conducts simulation analysis according to the actual phase demodulation method, which verifies the theory’s correctness. In addition, using this theory, it can be concluded that under the condition that the total size of the telescope aberration is determined to be stable, reducing the ratio of rotationally symmetric aberrations such as “spherical aberration” and “defocusing” among common aberrations can significantly improve the stability of the telescope. The conclusion guides the optical system design of LISA or Taiji.