With the rapid development of science and technology, ultrasound has been paid more and more attention by people, and it is widely used in engineering, diagnosis, and detection. In this paper, an ultrasonic image recognition method based on immune algorithm is proposed for ultrasonic images, and its method is applied to medical ultrasound liver image recognition. Firstly, this paper grays out the ultrasound liver image and selects the region of interest of the image. Secondly, it extracts the feature based on spatial gray matrix independent matrix, spatial frequency decomposition, and fractal features. Then, the immune algorithm is used to classify and identify the normal liver, liver cirrhosis, and liver cancer ultrasound images. Finally, based on the deficiency of the immune algorithm, it is combined with the support vector machine to form an optimized immune algorithm, which improves the performance of ultrasonic liver image classification and recognition. The simulation shows that this paper can effectively classify the normal liver, liver cirrhosis, and liver cancer ultrasound images. Compared with the traditional immune algorithm, this paper combines the immune algorithm with the support vector machine, and the optimized immune algorithm can effectively improve the performance of ultrasonic liver image classification and recognition.