To effectively respond to the COVID-19 pandemic, California had to quickly mobilize a substantial number of case investigators (CIs) and contact tracers (CTs). This workforce was comprised primarily of redirected civil servants with diverse educational and professional backgrounds. The purpose of this evaluation was to understand whether the weeklong, remote course developed to train California's CI/CT workforce (i.e., Virtual Training Academy) adequately prepared trainees for deployment. From May 2020 to February 2021, 8,141 individuals completed the training. A survey administered ~3 weeks post-course assessed two measures of overall preparedness: self-perceived interviewing proficiency and self-perceived job preparedness. Bivariate analyses were used to examine differences in preparedness scores by education level, career background, and whether trainees volunteered to join the COVID-19 workforce or were assigned by their employers. There were no significant differences in preparedness by education level. Compared to trainees from non-public health backgrounds, those from public health fields had higher self-perceived interviewing proficiency (25.1 vs. 23.3, p < 0.001) and job preparedness (25.7 vs. 24.0, p < 0.01). Compared to those who were assigned, those who volunteered to join the workforce had lower self-perceived job preparedness (23.8 vs. 24.9, p = 0.02). While there were some statistically significant differences by trainee characteristics, the practical significance was small (<2-point differences on 30-point composite scores), and it was notable that there were no differences by education level. Overall, this evaluation suggests that individuals without bachelor's degrees or health backgrounds can be rapidly trained and deployed to provide critical disease investigation capacity during public health emergencies.