The two types of nonlinear optical cryptosystems (NOCs) that are respectively based on amplitude-phase retrieval algorithm (APRA) and phase retrieval algorithm (PRA) have attracted a lot of attention due to their unique mechanism of encryption process and remarkable ability to resist common attacks. In this paper, for the first time, to the best of our knowledge, the security of the two types NOCs is evaluated by using a deep-learning (DL) method, wherein an end-to-end densely connected convolutional network (DenseNet) model for cryptanalysis is developed. The proposed DL-based method is able to retrieve unknown plaintexts from the given ciphertexts by using the trained DenseNet model without a prior knowledge of any public or private keys. The results of numerical experiments with the DenseNet model clearly demonstrate the validity and good performance of the proposed the DL-based attack on NOCs.