To solve the separation of multi-source signals and detect their features from a single channel, a signal separation method using multi-constraint non-negative matrix factorization (NMF) is proposed. In view of the existing NMF algorithm not performing well in the underdetermined blind source separation, the β-divergence constraints and determinant constraints are introduced in the NMF algorithm, which can enhance local feature information and reduce redundant components by constraining the objective function. In addition, the Sine-bell window function is selected as the processing method for short-time Fourier transform (STFT), and it can preserve the overall feature distribution of the original signal. The original vibration signal is first transformed into time–frequency domain with the STFT, which describes the local characteristic of the signal from the time–frequency distribution. Then, the multi-constraint NMF is applied to reduce the dimensionality of the data and separate feature components in the low dimensional space. Meanwhile, the parameter WK is constructed to filter the reconstructed signal that recombined with the feature component in the time domain. Ultimately, the separated signals will be subjected to envelope spectrum analysis to detect fault features. The simulated and experimental results indicate the effectiveness of the proposed approach, which can realize the separation of multi-source signals and their fault diagnosis of bearings. In addition, it is also confirmed that the proposed method, juxtaposed with the NMF algorithm of the traditional objective function, is more applicable for compound fault diagnosis of the rotating machinery.