Filter-based spectral beam combining (FSBC) is a promising power-scaling concept for high-power, broad-linewidth fiber lasers, as it relaxes the requirements for linewidth control and also the sizes of the individual beams. As the combining element in the FSBC system, the steep-edge filter plays a major role in achievement of the combining efficiency and the beam quality. In this case, we combine the uncorrelated surface roughness model and the combining efficiency model, and we conduct a comprehensive analysis of the effects of surface roughness, thickness error, and incident angle on the filter’s optical properties and the combining efficiency, in order to determine the optimal configuration for the laser beam-combining system. The simulation results show a good agreement with the measured ones. Meanwhile, through the adoption of the angular spectrum theory, this paper has also conducted a preliminary analysis of the influence of the combining elements on the quality of the combined beam, and some theoretical instructions on the future design of the spectral beam-combining system are provided.