For the optical signal analysis and processing of optical imaging with stray light beams, a new analysis method for smoothing the optical image edge and maintaining the spectrum signal well is presented in this paper. By calculating the mean gray value of the light pixel in the filter window of adaptive algorithm, we can analyze and compare the difference between the mean gray value and the current gray value with the light pixel of stray light beams, and can determine the smoothness of the optical image edge. This method can increase greatly the ability of optical image analysis and processing, and effectively reduce the false edge and the edge loss for the optical imaging with stray light beams, so that the optical image with high quality can be obtained.