Obesity and laziness are some of the common issues in the majority of the youth today. This has led to the development of a proposed exergaming solution where users can play first-person physical games. This research study not only proposes a solution for physical fitness in the form of a game using wearable sensors but also proposes a multi-purpose system that provides different applications when trained for the domain-specific dataset. Critical tasks of gesture recognition and depiction in virtual reality can be applied to many applications in the domains of crime detection, fitness, healthcare, online learning, and sports. In particular, the proposed system enables a user to perform, detect, and depict different gestures in the virtual reality game. First, the system pre-processes input data by applying a median filter to overcome the anomalies. Then, features are extracted through a convolutional neural network, power spectral density, skewness, and kurtosis methods. Further, the system optimizes different features by using the grey wolf optimization. Lastly, the feature set which is optimized is fed to a recurrent neural network for classification. When Compared to the traditional methods, the suggested system gives better results while being easier to use. The IMSporting behaviors (IMSB) dataset includes badminton and other physical activities, the WISDM dataset includes common locomotor motions, and the ERICA dataset which includes a variety of exercises, were used in the experimentation. According to experimental findings, the suggested approach outperformed current methods, which showed detection accuracies of 85.01%, 88.46%, and 93.18% over the IMSB, WISDM, and ERICA datasets, respectively.