Breast Cancer (BC) is a major health issue in women of the age group above 45. Identification of BC at an earlier stage is important to reduce the mortality rate. Image-based noninvasive methods are used for early detection and for providing appropriate treatment. Computer-Aided Diagnosis (CAD) schemes can support radiologists in making correct decisions. Computational intelligence paradigms such as Machine Learning (ML) and Deep Learning (DL) have been used in the recent past in CAD systems to accelerate diagnosis. ML techniques are feature driven and require a high amount of domain expertise. However, DL approaches make decisions directly from the image. The current advancement in DL approaches for early diagnosis of BC is the motivation behind this review. This article throws light on various types of CAD approaches used in BC detection and diagnosis. A survey on DL, Transfer Learning, and DL-based CAD approaches for the diagnosis of BC is presented in detail. A comparative study on techniques, datasets, and performance metrics used in state-of-the-art literature in BC diagnosis is also summarized. The proposed work provides a review of recent advancements in DL techniques for enhancing BC diagnosis.