In the process of extracting the characteristic frequency of the fault signal of the rolling bearing in the mechanical system, the signal transmission paths between the fault points inside the bearing and the sensor are diversified, which will produce compound faults of different forms and combinations. These composite faults will lead to a series of serious consequences such as distortion and aliasing of the fault characteristic spectrum. In order to solve these problems, this paper proposes a composite fault feature separation method for rolling bearing based on the mixed function decoupling model. Firstly, the functional series model of the fault system is established, and some kernel functions of functional series are obtained. Secondly, the coupling frequency relationship is established and the recursive search algorithm of frequency energy is applied to decouple the compound fault coupling frequency into a single fault frequency feature set. Finally, the threshold method of fault characteristic frequency energy entropy is utilized to optimize the single fault frequency feature set and identify the true and false features, so as to identify each characteristic frequency point of the composite fault. Experimental results show that, under the condition of low SNR, the proposed method can decouple no more than three typical composite fault signals without depending on the signal filter.