Purpose In medical image analysis, deep learning has great application potential. Discovering a method for extracting valuable information from medical images and integrating that information closely with medical treatment has recently become a major topic of interest. Because obtaining large volumes of breast lesion ultrasound image data is difficult, transfer learning is usually employed to obtain benign and malignant classification of breast lesions. However, because of blurred unclear regions of interest in breast lesion ultrasound images and severe speckle noise interference, convolutional neural networks have proven ineffective in extracting features, thus providing unreliable classification results. Methods This study employs image decomposition to obtain fuzzy enhanced and bilateral filtered images to enrich input information of breast lesions. Fuzzy enhanced, bilateral filtered, and original ultrasound images comprise multifeature data, which are presented as inputs to a pre‐trained model to realize knowledge fusion. Therefore, effective features of breast lesions are extracted and then used to train fully connected layers with ground truths provided by a doctor to accomplish the classification. Results A pre‐trained VGG16 model was used to extract features from multifeature data, and these features were fused to train the fully connected layers to realize classification. The performance score reported is as follows: accuracy of 93%, sensitivity of 95%, specificity of 88%, F1 score of 0.93, and AUC of 0.97. Conclusions Compared with using a single original ultrasound image for feature extraction, multifeature data based on image decomposition enables the pre‐trained model to extract more relevant features, thereby providing better classification results than those from traditional transfer learning techniques.