Breast cancer accounts for the highest number of female deaths worldwide. Early detection of the disease is essential to increase the chances of treatment and cure of patients. Infrared thermography has emerged as a promising technique for diagnosis of the disease due to its low cost and that it does not emit harmful radiation, and it gives good results when applied in young women. This work uses convolutional neural networks in a database of 440 infrared images of 88 patients, classifying them into two classes: normal and pathology. During the training of the networks, we use transfer learning of the following convolutional neural network architectures: AlexNet, GoogLeNet, ResNet-18, VGG-16, and VGG-19. Our results show the great potential of using deep learning techniques combined with infrared images in the aid of breast cancer diagnosis.