Gear break and pitting are two common faults in transmission system, when these two faults coexist and form a compound fault, the damage speed and frequency of gear transmission system will be greatly increased. Taking the gear fault-pitting compound fault as the object, the dynamic model of gear single fault and compound fault is established, and the vibration characteristics of gear single fault, pitting single fault and broken tooth-pitting compound fault signal are analyzed. The characteristic manifolds of the intrinsic dimension space in the case of gear single failure and compound fault are extracted by using the Laplacian Eigenmaps algorithm, the evolution trend of single fault and compound fault in the overlapping region of the feature space, the degree of correlation and the curvature of the fault circle core are analyzed and obtained. The study found that with the deepening of the fault severity, the overlapping area of fault circle between compound fault and single fault become smaller gradually, that is, the degree of correlation become weakened, tooth broken single fault and compound fault can be identified in mid-late stage of fault, while the pitting single fault and compound fault are in the late stage. The experimental results of gearbox compound fault correlation show that the conclusion of the simulation analysis is correct and effective, which provides a new idea for the diagnosis of mechanical complex faults.