Rail health monitoring plays an important role in the recent railway system. In order to implement acoustic emission technique in rail health monitoring, the mechanical noise caused by wheel-rail interactions should be eliminated with digital signal processing techniques in advance. This paper proposed a wavelet subband least mean square (LMS) adaptive filter, which can be used to effectively eliminate the strong noise and detect crack signals in railway. In such a method, the noisy input signals and reference noise should be first transformed into multi-scale wavelet coefficients. Then the decomposed noisy input and reference of the same channel was input into the separate LMS adaptive filters and the parameters of the filters were optimized by the metric of similarity coefficient. Finally, the denoised wavelet components in different levels were integrated into the whole denoised signal. Experimental tests clearly demonstrate that, compared to the conventional LMS method and adaptive wavelet filtering method, the proposed algorithm can improve the detectability, obtain a higher SNR, and at the same time retain the details of the crack signals under the actual railway noise interference.