A Novel Approach Using Non-Experts and Transformation Models to Predict the Performance of Experts in A/B Tests

Abstract

The European Union is committed to modernising and improving air traffic management systems to promote environmentally friendly air transport. However, the safety-critical nature of ATM systems requires rigorous user testing, which is hampered by the scarcity and high cost of air traffic controllers. In this article, we address this problem with a novel approach that involves non-experts in the evaluation of expert software in an A/B test setup. Using a transformation model that incorporates auxiliary information from a newly developed psychological questionnaire, we predict the performance of air traffic controllers with high accuracy based on the performance of students. The transformation model uses multiple linear regression and auxiliary information corrections. This study demonstrates the feasibility of using non-experts to test expert software, overcoming testing challenges and supporting user-centred design principles.