In recent years, the general public and the technology industry have favored stereoscopic vision, immersive experience, and real-time visual information reception of virtual reality (VR) and augmented reality (AR). The device carrier, the Head-Mounted Display (HMD), is recognized as one of the next generation’s most promising computing and communication platforms. HMD is a virtual image optical display device that combines optical lens modules and binocular displays. The visual impact it brings is much more complicated than the traditional display and also influences the performance of image quality. This research investigated the visual threshold of stray light for three kinds of VR HMD devices, and proposes a qualitative model, derived from psychophysical experiments and the measurement of images on VR devices. The recorded threshold data of the psychophysical stray light perception experiment was used as the target when training. VR display image captured by a wide-angle camera was processed, through a series of image processing procedures, to extract variables in the range of interest. The machine learning algorithm established an evaluation method for human eye-perceived stray light in the study. Four supervised learning algorithms, including K-Nearest Neighbor (KNN), Logistic Regression (LR), Support Vector Machine (SVM), and Random Forest (RF), were compared. The established model’s accuracy was about 90% in all four algorithms. It also proved that different percentages of thresholds could be used to label data according to demand to predict the feasibility of various subdivision inspection specifications in the future. This research aimed to provide a fast and effective stray light qualitative evaluation method to be used as a basis for future HMD optical system design and quality control. Thus, stray light evaluation will become one of the critical indicators of image quality and will be applicable to VR or AR content design.