We have developed a computer-aided diagnosis system based on a convolutional neural network that aims to classify breast mass lesions in optical tomographic images obtained using a diffuse optical tomography system, which is suitable for repeated measurements in mass screening. Sixty-three optical tomographic images were collected from women with dense breasts, and a dataset of 1260 2D gray scale images sliced from these 3D images was built. After image preprocessing and normalization, we tested the network on this dataset and obtained 0.80 specificity, 0.95 sensitivity, 90.2% accuracy, and 0.94 area under the receiver operating characteristic curve (AUC). Furthermore, a data augmentation method was implemented to alleviate the imbalance between benign and malignant samples in the dataset. The sensitivity, specificity, accuracy, and AUC of the classification on the augmented dataset were 0.88, 0.96, 93.3%, and 0.95, respectively.