PurposePredicting student performance is crucial in educational settings to identify and support students who may need additional help or resources. Understanding and predicting student performance is essential for educators to provide targeted support and guidance to students. By analyzing various factors like attendance, study habits, grades, and participation, teachers can gain insights into each student’s academic progress. This information helps them tailor their teaching methods to meet the individual needs of students, ensuring a more personalized and effective learning experience. By identifying patterns and trends in student performance, educators can intervene early to address any challenges and help students acrhieve their full potential. However, the complexity of human behavior and learning patterns makes it difficult to accurately forecast how a student will perform. Additionally, the availability and quality of data can vary, impacting the accuracy of predictions. Despite these obstacles, continuous improvement in data collection methods and the development of more robust predictive models can help address these challenges and enhance the accuracy and effectiveness of student performance predictions. However, the scalability of the existing models to different educational settings and student populations can be a hurdle. Ensuring that the models are adaptable and effective across diverse environments is crucial for their widespread use and impact. To implement a student’s performance-based learning recommendation scheme for predicting the student’s capabilities and suggesting better materials like papers, books, videos, and hyperlinks according to their needs. It enhances the performance of higher education.Design/methodology/approachThus, a predictive approach for student achievement is presented using deep learning. At the beginning, the data is accumulated from the standard database. Next, the collected data undergoes a stage where features are carefully selected using the Modified Red Deer Algorithm (MRDA). After that, the selected features are given to the Deep Ensemble Networks (DEnsNet), in which techniques such as Gated Recurrent Unit (GRU), Deep Conditional Random Field (DCRF), and Residual Long Short-Term Memory (Res-LSTM) are utilized for predicting the student performance. In this case, the parameters within the DEnsNet network are finely tuned by the MRDA algorithm. Finally, the results from the DEnsNet network are obtained using a superior method that delivers the final prediction outcome. Following that, the Adaptive Generative Adversarial Network (AGAN) is introduced for recommender systems, with these parameters optimally selected using the MRDA algorithm. Lastly, the method for predicting student performance is evaluated numerically and compared to traditional methods to demonstrate the effectiveness of the proposed approach.FindingsThe accuracy of the developed model is 7.66%, 9.91%, 5.3%, and 3.53% more than HHO-DEnsNet, ROA-DEnsNet, GTO-DEnsNet, and AOA-DEnsNet for dataset-1, and 7.18%, 7.54%, 5.43% and 3% enhanced than HHO-DEnsNet, ROA-DEnsNet, GTO-DEnsNet, and AOA-DEnsNet for dataset-2.Originality/valueThe developed model recommends the appropriate learning materials within a short period to improve student’s learning ability.