White Light Interferometry is a highly precise non-contact measurement technology. However, in practical applications, certain noise sources such as thermal noise, shot noise, and redundant intensity noise, can introduce signal distortion. In this paper, we propose a mathematical formulation for the key metric, Quality Factor, which serves as an evaluation criterion for the quality of interference fringes. The influence of different factors on the measurement is quantitatively analyzed using the Signal-to-Noise Ratio (SNR), and various filtering methods are employed to address different types of noise. Finally, experimental results highlight the primary factors and optimization techniques that significantly impact the quality of interference fringes, thereby verifing the effectiveness of the method on measurement results.