Pork floss is a common dried meat product in Asia. The endpoint of the baking process is traditionally determined by subjective human experts and indirect temperature measurements, which can often result in unstandardized production. Current colorimeters are unavailable for onsite measurement due to limitations associated with contact measurement and environmental temperature. Instead of the abovementioned human experts and tabletop colorimeters, a handheld colorimeter was built based on the expertise of human specialists and utilizing a tabletop colorimeter and other optical steps. First, the selected samples were used to determine the upper and lower limits distinguishing light, medium, and heavy baking levels by using a tabletop colorimeter. Second, independent light sources and spectrometers were utilized to choose the characteristic and reference wavelengths at 450 and 830 nm, separately. Third, the handheld colorimeter, instead of human expert observation, was designed with functions such as distance sensing and Internet of Things capabilities. The baked index was derived from the calibration reflection and established statistical models. Here, the calibration reflection was defined by the normalized intensity at 450 nm relative to 830 nm, and statistical models were founded from the determined samples of upper and lower limits at 95–700 mm. The developed handheld colorimeter demonstrated high agreement rates of 96.84% and 93.86% in separate comparisons with tabletop colorimeters and human experts, respectively. This work indicated the accurate and stable recognition of samples within two limits and overall. Field validation confirmed the performance of remote, economic, and onsite recognition against environmental temperature and noise.