Breast cancer is one of the deadliest cancer types affecting women worldwide. As with all types of cancer, early detection of breast cancer is of vital importance. Early diagnosis plays an important role in reducing deaths and fighting cancer. Ultrasound (US) imaging is a painless and common technique used in the early detection of breast cancer. In this article, deep learning-based approaches for the classification of breast US images have been extensively reviewed. Classification performance of breast US images of architectures such as AlexNet, VGG, ResNet, GoogleNet and EfficientNet, which are among the most basic CNN architectures, has been compared. Then, transformer models, which are one of the most popular deep learning architectures these days and show similar performance to the performance of CNN' architectures in medical images, are examined. BUSI, the only publicly available dataset, was used in experimental studies. Experimental studies have shown that the transformer and CNN models successfully classify US images of the breast. It has been observed that vision transformer model outperforms other models with 88.6% accuracy, 90.1% precison, 87.4% recall and 88.7% F1-score. This study shows that deep learning architectures are successful in classification of US images and can be used in the clinic experiments in the near future.