Faults in bearings and gearboxes, which are widely used in rotating machines, can lead to heavy investment and productivity loss. Accordingly, a fault diagnosis system is necessary to ensure a high-performance transmission. However, as mechanical fault diagnosis enters the era of big data, it can be difficult to apply traditional fault diagnosis methods because of the massive computation cost and excessive reliance on human labor. Meanwhile, unsupervised learning has been shown to have excellent performance in processing machanical big data. In this paper, an unsupervised learning method known as sparse filtering is applied, the multi-correlation of a weight matrix is investigated, and a method that is more suitable for the feature extraction of signals is proposed. The main contribution of our work is the modification of original method. First, to understand the non-monotonicity testing accuracies of the original method, the physical interpretation of input dimensions is studied. Second, using the physical interpretation, an overfitting phenomenon is discovered and examined. Third, to reduce the overfitting phenomenon, a method which eliminates the multi-correlation of the weight matrix is proposed. Finally, bearing and gear datasets are employed to verify the effectiveness of the proposed method; experimental results show that the proposed method can achieve a superior performance in comparison to the original sparse filtering model.