Diabetic retinopathy (DR) is an important cause of blindness worldwide. However, DR is hard to be detected in the early stages, and the diagnostic procedure can be time-consuming even for the experienced experts. Therefore, a computer-aided diagnosis method based on deep learning algorithms is proposed to automatedly diagnose the referable diabetic retinopathy by classifying color retinal fundus photographs into two grades. In this paper, a novel convolutional neural network model with the Siamese-like architecture is trained with a transfer learning technique. Different from the previous works, the proposed model accepts binocular fundus images as inputs and learns their correlation to help to make a prediction. In the case with a training set of only 28 104 images and a test set of 7024 images, an area under the receiver operating curve of 0.951 is obtained by the proposed binocular model, which is 0.011 higher than that obtained by the existing monocular model. To further verify the effectiveness of the binocular design, a binocular model for five-class DR detection is also trained and evaluated on a 10% validation set. The result shows that it achieves a kappa score of 0.829 which is higher than that of the existing non-ensemble model.INDEX TERMS Biomedical imaging processing, diabetic retinopathy, fundus photograph, convolutional neural network, deep learning, Siamese-like network.