Breast cancer is one of the most common types of cancer among women, which requires building smart systems to help doctors and early detection of cancer. Deep learning applications have emerged in many fields, especially in health care, but there are still some limitations in this technique, such as the small number of classified medical images needed to train deep learning models, to solve this problem, transfer learning technology appeared based on transferring knowledge from pre-trained models on ImageNet and setting them to the current task, but there is still a problem that the features extracted from ImageNet are not medical, so this paper aims to find a solution to this problem by using new techniques for transfer learning by taking advantage of the presence of unclassified medical images of the same disease to reduce the impact of ImageNet. The proposed approach was applied to the modified Xception model to classify the histological images of breast cancer in the ICIAR 2018 dataset into four classes: invasive carcinoma, in situ carcinoma, benign, and normal. The proposed approach has obtained 99%, 99.003%, 98.995%, 99%, 98.55%, and 99.14% for accuracy, precision, recall, F1-score, sensitivity, and specificity respectively. Our work has been compared with previous works.