Finger-Knuckle-Print (FKP) is an accurate and reliable biometric in compare to other hand-based biometrics like fingerprint because of the finger's dorsal region is not exposed to surfaces. In this paper, a simple end-to-end method based on Convolutional Neural Network (CNN) is proposed for FKP recognition. The proposed model is composed only of three convolutional layers and two fully connected layers. The number of trainable parameters hereby has significantly reduced. Additionally, a straightforward method is utilized for data augmentation in this paper. The performance of the proposed network is evaluated on Poly-U FKP dataset based on 10-fold cross-validation. The best recognition accuracy, mean accuracy and standard deviation are 99.83%, 99.18%, and 0.76, respectively. Experimental results show that the proposed method outperforms the state-of-the-arts in terms of recognition accuracy and the number of trainable parameters. Also, in compare to four fine-tuned CNN models including AlexNet, VGG16, ResNet34, and GoogleNet, the proposed simple method achieved higher performance in terms of recognition accuracy and the numbers of trainable parameters and training time.