PurposeThe quality of ultrasound images is degraded by speckle and Gaussian noises. This study aims to develop a deep‐learning (DL)‐based filter for ultrasound image denoising.MethodsA novel DL‐based filter using adaptive residual (AdaRes) learning was proposed. Five image quality metrics (IQMs) and 27 radiomics features were used to evaluate denoising results. The effect of our proposed filter, AdaRes, on four pre‐trained convolutional neural network (CNN) classification models and three radiologists was assessed.ResultsAdaRes filter was tested on both natural and ultrasound image databases. IQMs results indicate that AdaRes could remove noises in three different noise levels with the highest performances. In addition, a radiomics study proved that AdaRes did not distort tissue textures and it could preserve most radiomics features. AdaRes could also improve the performance classification using CNNs in different settings. Finally, AdaRes also improved the mean overall performance (AUC) of three radiologists from 0.494 to 0.702 in the classification of benign and malignant lesions.ConclusionsAdaRes filtered out noises on ultrasound images more effectively and can be used as an auxiliary preprocessing step in computer‐aided diagnosis systems. Radiologists may use it to remove unwanted noises and improve the ultrasound image quality before the interpretation.