IntroductionDespite the well‐known reproducibility issues of subjective refraction, most studies evaluating autorefractors compared differences between the device and subjective refraction. This work evaluated the performance of a novel handheld Hartmann–Shack‐based autorefractor using an alternative protocol, which considered the inherent variability of subjective refraction.MethodsParticipants underwent an initial measurement with a desktop autorefractor, two subjective refractions (SR1 and SR2) and a final measurement with the QuickSee Free (QSFree) portable autorefractor. Autorefractor performance was evaluated by comparing the differences between the QSFree and each of the subjective refractions with the difference between the subjective refractions (SR1 vs. SR2) using Bland–Altman analysis and percentage of agreement.ResultsA total of 75 subjects (53 ± 14 years) were enrolled in the study. The average difference in the absolute spherical equivalent (M) between the QSFree and the SR1 and SR2 was ±0.24 and ±0.02 D, respectively, that is, very similar or smaller than the SR1 versus SR2 difference (±0.26 D). Average differences in astigmatic components were found to be negligible. The results demonstrate that differences between QSFree and both subjective refractions in J0 and J45 were within ±0.50 D for at least 96% of the measurements. The limits of agreement (LOAs) of the differences between QSFree and SR1, as well as QSFree and SR2, were higher than those observed between SR1 and SR2 for M, J0 and J45.ConclusionsA protocol was designed and validated for the evaluation of a refractive device to account for the variability of subjective refraction. This protocol was used to evaluate a novel portable autorefractor and observed a smaller difference between the device and subjective refractions than the difference between the two subjective refraction measurements in terms of mean bias error, although the standard deviation was higher.