Unbalanced mass identification is important for rotor systems. Current methods normally use sensors, which only detect vibration in two-dimensional (2D) space. Actually, the rotor systems vibrate in three-dimensional directions. In this paper, a non-contact method is developed to identify unbalanced mass of rotor systems in 3D space. A stereo video system with a pair of synchronized high-speed cameras is established and a feature point is employed to replace traditional contact transducer for measurement. Checkerboard target on a vibration table is used to implement dynamic calibration. The proposed method is compared with eddy current method and laser displacement method. The comparison experiments verify the detection ability of the unbalanced mass for the proposed method. Overall, the proposed method can provide more information than 2D detection methods, which has the great potential for fault diagnosis of rotating machinery.