IntroductionStudent success in Science, Technology, Engineering, and Mathematics (STEM) is a national concern. To increase engineering retention and graduation rates at a small private institution, a university council developed a binary classifier to identify high-risk students and proposed interventions that included decoupling first-year Physics and Calculus courses, support in introductory Calculus, and Spatial Visualization (SV) training. This paper aims to validate the binary classifier used to identify the under-prepared students entering their first year and assess the impact of the interventions. We provide a comparative analysis of student success metrics for high-risk engineering students across a decade of cohorts, including 5 years before (2006–2010) and 5 years after (2011–2015) implementation of intentional strategies.MethodsWe validated the binary classifier using an accuracy measure and Matthews Correlation Coefficient (MCC). We used the 2-population proportion test to compare STEM retention and 4- and 6-year graduation rates of High-Risk engineering students before and after interventions and compare student performance in early foundation STEM courses across the same time frame.ResultsThe binary classification model identified High-Risk students with an accuracy of 63–70% and an MCC of +0.28 to +0.30. In addition, we found statistically significant improvement (p < 0.001) in the STEM retention rates, 6-year graduation rates, and first part of Physics, Calculus, and Chemistry sequences after the interventions.DiscussionThe methodology and strategies presented may provide effective guidance for institutions seeking to improve the overall performance of undergraduate students who otherwise might struggle in their first-year engineering curriculum.