Novelty detection has been developed into a state-of-the-art technique to detect abnormal behavior and trigger alarm for in-field machine maintenance. With built-up models of normality, it has been widely applied to several situations with normal supervising dataset such as shaft rotating speed and component temperature available meanwhile in the absence of fault information. However, the research about vibration transmission based novelty detection remains unnoticed until recently. In this paper, vibration transmission measurement on rotor is performed; based on extreme value distributions, thresholds for novelty detection are calculated. In order to further decrease the false alarm rate, both measurement and segmentation uncertainty are considered, as they may affect threshold value and detection correctness heavily. Feasible reduction strategies are proposed and discussed. It is found that the associated multifractal coefficient and Kullback-Leibler Divergence operate well in the uncertainty reduction process. As shown by in situ applications to abnormal rotor with pedestal looseness, it is demonstrated that the abnormal states are detected. The higher specificity value proves the effectiveness of proposed uncertainty reduction method. This paper shows novel achievements of uncertainty reduced novelty detection applied to vibration signal in dynamical system and also sheds lights on its utilization in the field of health monitoring of rotating machinery.