Anxiety is an important issue that affects their academic performance, mental health, and overall educational journey. To address this issue, it is important to accurately assess anxiety levels and provide evidence-based techniques. However, due to the complexity of anxiety and individual differences, analyzing clustering algorithms to efficiently classify psychological levels is challenging. Traditional clustering techniques face certain challenges in accurately classifying anxiety levels, such as slow convergence, sensitivity to initial conditions, and difficulties in handling constraints. To address these issues, clustering with an improved Mayfly-based optimization algorithm (IMOA) is proposed based on the dynamic variable for better performance to classify psychological levels. Initially, IMOA is validated using 23 standard benchmark functions, confirming its ability to find optimal solutions. Then, IMOA is applied to the student dataset, classifying them into Cluster A and Cluster B. The average scores for both clusters across all test cases are 76.7% and 53.07%, respectively. These results demonstrate the formation of dissimilar student groups with homogeneous emotions and performance, highlighting the importance of addressing emotional stress. Finally, by assigning students to clusters, educators and mental health professionals can better support those who may struggle, ensuring they receive the attention and resources they need. The obtained results show that IMOA with a dynamic variable effectively classifies student anxiety, improving the learning environment and helping teachers better understand students’ needs. This identification allows them to provide more effective support and adapt their teaching to meet the specific needs of those seeking support.