User experience evaluation for MIDRC AI interface

Abstract

Building Machine Learning models from scratch for clinical applications can be a challenging undertaking requiring varied levels of expertise. Given the heterogeneous nature of input data and specific task requirements, even seasoned developers and researchers may occasionally run into issues with incompatible frameworks. This is further complicated in the context of diagnostic radiology. Therefore, we developed the CRP10 AI Application Interface (CRP10AII) as a component of the Medical Imaging and Data Resource Center (MIDRC) to deliver a modular and user-friendly software solution that can efficiently address the demands of physicians, early AI developers to explore, train, and test AI algorithms. 37 The CRP10AII tool is python-based web framework that is connected to the data commons (GEN3) that offers the ability to develop AI models from scratch or employ pre-trained models while allowing for visualization and interpretation of the predictions of the AI model. Here, we evaluate the capabilities of CRP10AII and its related human-API interaction factors. This evaluation aims at investigating various aspects of the API, including:(i) robustness and ease of use; (ii) visualization help in decision making tasks; and (iii) necessary further improvements for initial AI researchers with different medical imaging and AI expertise levels. Users initially experienced trouble testing the API; however, the problems have since been fixed as a result of additional explanations. The user evaluation's findings demonstrate that although different options on the API are generally easy to understand, use, and helpful in decision-making tasks for users with and without experience in medical imaging and AI, there are differences in how the various options are understood and used by users. We were also able to collect additional inputs, such as increasing information fields and including more interactive components to make the API more generalizable and customizable.