Administering exams in large enrollment courses is challenging and systems in place for accomplishing this task were upended in the spring of 2020 when a sudden transformation to online instruction and testing occurred due to the COVID-19 pandemic. In the following year, when courses remained online, approaches to improve exam security included measures like using test banks and reducing the allotted time for completing exams to reduce the sharing of information. A psychometric comparison using classical test theory of an unproctored online exam with one delivered in-person indicates both have comparable reliability. However, item-level analysis demonstrates some questions performed higher in the unproctored setting, with an important variable being the item’s searchability online. Revising questions to increase generalizability and reduce searchability mitigate these performance differences. Further, changing the format for questions involving calculations from multiple-choice to open response with random number generation was found to increase difficulty and item discrimination and is a viable approach for improving exam security.