Accurate prediction of student performance is crucial in learning analytics to prevent course failures and improve academic outcomes. However, publicly accessible educational data often contains noise and imbalanced data distributions, requiring effective handling techniques. In this study, we propose a novel approach that combines the Synthetic Minority Over-sampling Technique (SMOTE) with Long Short-Term Memory (LSTM) and Feed-Forward Neural Network (FFNN) models for performance prediction in virtual learning environments (VLEs). Our experimental results show that utilizing the SMOTE technique significantly improves the accuracy of predicting student withdrawals, with the LSTM model achieving the highest accuracy of 94.90% in the 25th week of data testing. These findings indicate the effectiveness of the SMOTE technique in addressing data imbalance issues in VLE datasets and the potential of our proposed deep learning models in accurately predicting student performance. The implications of our study are significant for learning analytics and educational institutions, as accurate prediction of student performance can inform early interventions and personalized support. Future research could explore the generalizability of our approach in diverse educational contexts and the integration of additional features for further improving prediction accuracy. Hence, our study contributes to the field of learning analytics by proposing a novel approach that combines SMOTE with deep learning models for student performance prediction in VLEs. Our findings highlight the potential of our approach in addressing data imbalance challenges and accurately predicting student performance, with implications for enhancing student success in educational settings.