With the development of high-speed rotating machinery, the unbalance vibration feature of each rotor system has a greater influence on the work efficiency, bearing life, operational time, etc. Therefore, an active balancing system is necessary to automatically reduce the unbalance vibration in the process of rotor operation. This study introduced a novel liquid transfer active balancing system for the hollow rotors, and compensation mass was performed by balance liquid transmission between two pairs of contra-positioned chambers. The performance of this new balance actuator was analyzed, including balancing velocity, balancing accuracy, and the effect on rotor dynamics. A monitoring and control program was constructed to control the balance actuator. Two extraction methods of synchronic vibration were introduced and quantitatively compared through simulation. A control program was developed and the control accuracy was within 1 ms. Furthermore, the effectiveness of the new balancing system was verified through active balance experiments and the maximum speed was 15,600 rpm. The results indicate that the balancing system could effectively decrease the unbalance vibration of the rotor system within 10 s, and the amount of decrease was more than 80%.