The detection and recognition of handwritten content is the process of converting non-intelligent information such as images into machine edit-able text. This research domain has become an active research area due to vast applications in a number of fields such as handwritten filing of forms or documents in banks, exam form filled by students, users' authentication applications. Generally, the handwritten content recognition process consists of four steps: data preprocessing, segmentation, the feature extraction and selection, application of supervised learning algorithms. In this paper, a detailed survey of existing techniques used for Hand Written Digit Recognition(HWDR) is carried out. This review is novel as it is focused on HWDR and also it only discusses the application of Neural Network (NN) and its modified algorithms. We discuss an overview of NN and different algorithms which have been adopted from NN. In addition, this research study presents a detailed survey of the use of NN and its variants for digit recognition. Each existing work, we elaborate its steps, novelty, use of dataset and advantages and limitations as well. Moreover, we present a Scientometric analysis of HWDR which presents top journals and sources of research content in this research domain. We also present research challenges and potential future work.