Effect of Machine Learning Cross-validation Algorithms Considering Human Participants and Time-series: Application on Biometric Data Obtained from a Virtual Reality Experiment

Abstract

Data containing human participants and time-series features, as is commonplace in Human Factors research, require special considerations when used in machine learning applications. Ignoring such features during cross-validation procedures might lead to artificially increased model performances due to temporal (i.e. using future observations to predict the present) and participant (i.e using sub-data sets coming from the same participant for training and testing) data leakage. We propose a comparison approach to assess the model performance when machine learning algorithms are trained with two distinctly different cross-validation algorithms: k-fold, which assumes data independence, and population-informed forward chain (PIFC), which accounts for human participants and time-series features. A case study was conducted by using biometric measurements collected from a virtual reality chess experiment. The results show that substantial overestimation might occur when applying the k-fold algorithm instead of the PIFC algorithm.

An Eye Tracking Based Framework for Safety Improvement of Offshore Operations

“…This is planned studies as well. Furthermore, upcoming studies aim to modify advanced techniques for analyzing eye movements, which encompass visual entropy ( 39 ), a multimodal approach to analyzing fatigue and eye movements ( 38 ), a framework for analyzing eye movements in real-time ( 45 ), and machine learning ( 44 ). These techniques can be utilized to formulate real-time safety assessment algorithms for various drilling operations at drilling rigs and Real Time Operation Centers.…”