These methods that rely on features and indices derived from nonlinear vibration transmissibility functions (NVTFs) have found widespread success in detecting potential bolt-loosening faults within nonrotating systems such as bridges, railways, and satellites. To improve and extend existing NVTF-based methods for diagnosing bolt-loosening faults in rotor systems, a novel method is proposed with detailed theoretical analysis and experimental study in this paper. First, a general rotor dynamic model, considering bolt-loosening forces, radial unbalance forces, and nonlinear support forces, is built for the nonlinear rotor dynamic analysis and the definition of rotor-domain NVTFs. Importantly, by dividing the rotor system into a series of rotor subsystems and analyzing corresponding rotor dynamic submodels to be diagnosed only, relationships between NVTFs and bolt-loosening forces are summarized, and then three sensitive fault features are defined. Based on this, local diagnosis indexes are developed, and a novel method with detailed operating flowchart is proposed accordingly. Finally, results from experimental cases on a testing rotor system with single/multiple stage bolt-loosening faults and loosened pedestals verify and demonstrate the effectiveness of the novel method. The study in this article successfully improves and extends existing NVTF-based methods for nonrotating systems to diagnose potential bolt-loosening faults in rotor systems even with nonlinear supports such as loosened pedestals.