Understanding the fundamental light-sample interaction process is a crucial step toward the development of vibrational spectroscopy to determine fruit texture (i.e., firmness). This study aimed to investigate the effect of pectin constitution, including total pectin, water-soluble pectin, protopectin contents, and protopectin index (PI), on the optical properties and firmness of ‘Baifeng’ and ‘Xiahui 8’ peach flesh at the different softening degrees during postharvest storage of 6 days at 20 °C. The firmness of ‘Baifeng’ and ‘Xiahui 8’ peaches significantly (p < 0.05) changed with a decreasing rate from 90.3% to 92.2%. Peach firmness of these two cultivars correlated well with PI contents (r > 0.912) and showed good internal correlations with optical scattering properties. The light absorption coefficient (μa) and reduced scattering coefficient (μ’s) at 600–1600 nm were measured using a single integrating sphere system combined with an inversion algorithm. This relationship of μa and μ’s with peach firmness and pectin constitution was first analyzed. Notably, the specific μ’s at 660 nm, 950 nm, 1203 nm, and 1453 nm showed a satisfactory prediction of peach firmness and PI of ‘Xiahui 8’ (R2 ≥ 0.926) and ‘Baifeng’ peaches (R2 ≥ 0.764), respectively. Furthermore, the prediction models were established based on partial least squares regression coupled with optical properties, and considerable prediction performances were obtained for tissue firmness (Rp2 ≥ 0.863) and PI based on μ’s (Rp2 ≥ 0.802). Consequently, these results further verified that the spectroscopic prediction model for peach firmness could be related to the high correlations between PI in tissues and their optical scattering properties. Future research interests could include the development of optical absorption and scattering sensors for rapid and efficient determination of peach firmness.