Breast cancer is the most common form of cancer in women. Its aggressive nature has made it one of the chief factors of high female mortality. Therefore, this has motivated research to achieve early diagnosis since it is the best strategy for patient survival. Currently, mammography is the gold standard for detecting breast cancer. However, it is expensive, unsuitable for dense breasts, and an invasive process that exposes the patient to radiation. Infrared thermography is gaining popularity as a screening modality for the early detection of breast cancer. It is a noninvasive and cost-effective modality that allows health practitioners to observe the temperature profile of the breast region for signs of cancerous tumors. Deep learning has emerged as a powerful computational tool for the early detection of breast cancer in radiology. As such, this study presents a review that shows existing work on deep learning-based Computer-aided Diagnosis (CADx) systems for breast cancer detection. In the same context, it reflects on classification utilizing breast thermograms. It first provides an overview of infrared thermography, details on available breast thermogram datasets, and then segmentation techniques applied to these thermograms. We also provide a brief overview of deep neural networks. Finally, it reviews works adopting Deep Neural Networks (DNNs) for breast thermogram classification.