Handwritten digit recognition (HDR) shows a significant application in the area of information processing. However, correct recognition of such characters from images is a complicated task due to immense variations in the writing style of people. Moreover, the occurrence of several image artifacts like the existence of intensity variations, blurring, and noise complicates this process. In the proposed method, we have tried to overcome the aforementioned limitations by introducing a deep learning- (DL-) based technique, namely, EfficientDet-D4, for numeral categorization. Initially, the input images are annotated to exactly show the region of interest (ROI). In the next phase, these images are used to train the EfficientNet-B4-based EfficientDet-D4 model to detect and categorize the numerals into their respective classes from zero to nine. We have tested the proposed model over the MNIST dataset to demonstrate its efficacy and attained an average accuracy value of 99.83%. Furthermore, we have accomplished the cross-dataset evaluation on the USPS database and achieved an accuracy value of 99.10%. Both the visual and reported experimental results show that our method can accurately classify the HDR from images even with the varying writing style and under the presence of various sample artifacts like noise, blurring, chrominance, position, and size variations of numerals. Moreover, the introduced approach is capable of generalizing well to unseen cases which confirms that the EfficientDet-D4 model is an effective solution to numeral recognition.