Background estimation is an efficient infrared (IR) small target detection method. However, to deal with unknown targets, the estimation window in existing algorithms should be adjusted to perform multiscale detection, and requires a lot of calculations. Besides, the stages during and after estimation have received wide attention in existing algorithms, but the research on the stages before estimation is insufficient. Moreover, existing algorithms typically regard the maximum value of different orientations as the estimation value. However, when a dim target is adjacent to high brightness background, it is easily submerged. This paper proposes a three-layer estimation window to detect targets of different sizes with only a single-scale calculation. The enhanced closest-mean background estimation method is then proposed and carefully designed before, during and after the estimation. Before estimation, the matched filter is adopted to improve the image signal-to-noise ratio. During estimation, the principle of closest-mean is proposed to suppress high brightness background. After estimation, a ratio-difference operation is performed to enhance the true target and suppress the background simultaneously. A simple checking mechanism is proposed to further improve the detection performance. Experiments on some IR images demonstrate the effectiveness and robustness of the proposed method. Compared with existing algorithms, the proposed method has better target enhancement, background suppression, and computational efficiency. Index Terms-IR small target, background estimation, matched filter, closest-mean, three-layer window.