The lifetime of wire rope is crucial in industry manufacturing, mining, and so on. The damage can be detected by using appropriate nondestructive testing techniques or destructive tests by cutting the part. For broken wires classification problems, this work is aimed at improving the recognition accuracy. Facing the defects at the exterior of the rope, a novel method for recognition of broken wires is firstly developed based on magnetic and infrared information fusion. A denoising method, which is adopted for magnetic signal, is proposed for eliminating baseline signal and wave strand. An image segmentation method is employed for parting the defects of infrared images. Characteristic vectors are extracted from magnetic images and infrared images, then kernel extreme learning machine network is applied to implement recognition of broken wires. Experimental results show that the denoising method and image segmentation are effective and the information fusion can improve the classification accuracy, which can provide useful information for estimating the residual lifetime of wire rope.