The broad application area and accompanying challenges make machine learning-based recognition of handwritten scripts a demanding field. Individuals' writing practices and inherent variations in the size, shape, and tilt of characters may increase the difficulty level. Deep convolutional neural network (DCNN) models have been successful in solving pattern recognition problems, but at the expense of a considerable number of trainable parameters and heavy computational loads. The proposed work addresses these problems by using the shifted window (SWIN) transformer method to recognize handwritten Devanagari numerals for the first time. In the presented model, the SWIN transformer is finely tuned to withstand popular DCNN models, such as VGG-16Net, ResNet-50, and DenseNet-121, in terms of recognition accuracy, space requirement, and computational complexity. The model successfully attained a recognition accuracy of 99.20% with only 0.218 million trainable parameters and 0.0912 giga floating-point operations per second (FLOPs). This indicates the validity and soundness of the proposed model for recognizing handwritten Devanagari numerals.