Breast cancer poses the greatest long-term health risk to women worldwide, in both industrialized and developing nations. Early detection of breast cancer allows for treatment to begin before the disease has a chance to spread to other parts of the body. The Internet of Things (IoT) allows for automated analysis and classification of medical pictures, allowing for quicker and more effective data processing. Nevertheless, Fog computing principles should be used instead of Cloud computing concepts alone to provide rapid responses while still meeting the requirements for low latency, energy consumption, security, and privacy. In this paper, we present CanDiag, an approach to cancer diagnosis based on Transfer Deep Learning (TDL) that makes use of Fog computing. This paper details an automated, real-time approach to diagnosing breast cancer using deep learning (DL) and mammography pictures from the Mammographic Image Analysis Society (MIAS) library. To obtain better prediction results, transfer learning (TL) techniques such as GoogleNet, ResNet50, ResNet101, InceptionV3, AlexNet, VGG16, and VGG19 were combined with the well-known DL approach of the convolutional neural network (CNN). The feature reduction technique principal component analysis (PCA) and the classifier support vector machine (SVM) were also applied with these TDLs. Detailed simulations were run to assess seven performance and seven network metrics to prove the viability of the proposed approach. This study on an enormous dataset of mammography images categorized as normal and abnormal, respectively, achieved an accuracy, MCR, precision, sensitivity, specificity, f1-score, and MCC of 99.01%, 0.99%, 98.89%, 99.86%, 95.85%, 99.37%, and 97.02%, outperforming some previous studies based on mammography images. It can be shown from the trials that the inclusion of the Fog computing concepts empowers the system by reducing the load on centralized servers, increasing productivity, and maintaining the security and integrity of patient data.