Background and AimsMental health problem is a rising public health concern. People of all ages, specially Bangladeshi university students, are more affected by this burden. Thus, the objective of the study was to use tree‐based machine learning (ML) models to identify major risk factors and predict anxiety, depression, and insomnia in university students.MethodsA social media‐based cross‐sectional survey was employed for data collection. We used Generalized Anxiety Disorder (GAD‐7), Patient Health Questionnaire (PHQ‐9) and Insomnia Severity Index (ISI‐7) scale for measuring students' anxiety, depression and insomnia problems. The tree‐based supervised decision tree (DT), random forest (RF) and robust eXtreme Gradient Boosting (XGBoost) ML algorithms were used to build the prediction models and their predictive performance was evaluated using confusion matrix and receiver operating characteristic (ROC) curves.ResultsOf the 1250 students surveyed, 64.7% were male and 35.3% were female. The students' ages ranged from 18 to 26 years old, with an average age of 22.24 years (SD = 1.30). Majority of the students (72.6%) were from rural areas and social media addicted (56.6%). Almost 83.3% of the students had moderate to severe anxiety, 84.7% had moderate to severe depression and 76.5% had moderate to severe insomnia problems. Students' social media addiction, age, academic performance, smoking status, monthly family income and morningness‐eveningness are the main risk factors of anxiety, depression and insomnia. The highest predictive performance was observed from the XGBoost model for anxiety, depression and insomnia.ConclusionThe study findings offer valuable insights for stakeholders, families and policymakers enabling a more profound comprehension of the pressing mental health disorders. This understanding can guide the formulation of improved policy strategies, initiatives for mental health promotion, and the development of effective counseling services within university campus. Additionally, our proposed model might play a critical role in diagnosing and predicting mental health problems among Bangladeshi university students and similar settings.