The vibration of a planetary gearbox (PG) is complex and mutually modulated, which makes the weak features of incipient fault difficult to detect. To target this problem, a novel method, based on an adaptive order bispectrum slice (AOBS) and the fault characteristics energy ratio (FCER), is proposed. The order bispectrum (OB) method has shown its effectiveness in the feature extraction of bearings and fixed-shaft gearboxes. However, the effectiveness of the PG still needs to be explored. The FCER is developed to sum up the fault information, which is scattered by mutual modulation. In this method, the raw vibration signal is firstly converted to that in the angle domain. Secondly, the characteristic slice of AOBS is extracted. Different from the conventional OB method, the AOBS is extracted by searching for a characteristic carrier frequency adaptively in the sensitive range of signal coupling. Finally, the FCER is summed up and calculated from the fault features that were dispersed in the characteristic slice. Experimental data was processed, using both the AOBS-FCER method, and the method that combines order spectrum analysis with sideband energy ratio (OSA-SER), respectively. Results indicated that the new method is effective in incipient fault feature extraction, compared with the methods of OB and OSA-SER.