Assessing the contribution of cochlear synaptopathy (CS) to the variability in speech-in-noise intelligibility among individuals remains a challenge. While several studies have proposed biomarkers for CS based on neural phase-locking to the temporal envelope (ENV), fewer have investigated how CS affects the coding of temporal fine structure (TFS), despite its crucial role in speech-in-noise perception. In this study, we specifically examined whether TFS-based markers of CS could be derived from electrophysiological responses and psychophysical detection thresholds of spectral modulation (SM) in a complex tone, which serves as a parametric model of speech. We employed an integrated approach, combining psychophysical testing with frequency-following response (FFR) measurements in three groups of participants: young normal-hearing (yNH), older normal-hearing (oNH), and older hearing-impaired (oHI) individuals. We expanded on previous work by assessing phase-locking to both ENV, using a 4 kHz rectangular amplitude-modulated (RAM) tone, and TFS, using a low-frequency (<1.5 kHz) SM complex tone. Overall, FFR results showed significant reductions in neural phase-locking to both ENV and TFS components with age and hearing loss. Specifically, the strength of TFS-related FFRs, particularly the component corresponding to the harmonic closest to the peak of the spectral envelope (~500 Hz), was negatively correlated with age, even after adjusting for audiometric thresholds. This TFS marker also correlated with ENV-related FFRs derived from the RAM tone, suggesting a shared decline in phase-locking capacity across low and high cochlear frequencies. Computational simulations of the auditory periphery indicated that the observed FFR strength reduction with age is consistent with approximately 50% loss of auditory nerve fibers, aligning with histopathological data. However, the TFS-based FFR marker did not account for variability in speech intelligibility observed in the same participants. Psychophysical measurements showed no age-related effects and were unrelated to the TFS-based FFR marker, highlighting the need for further psychophysical research to establish a behavioral counterpart. Altogether, our results demonstrate that FFRs to vowel-like stimuli can serve as a complementary electrophysiological marker for assessing neural coding fidelity to stimulus TFS. This approach could provide a valuable tool for better understanding the impact of CS on an important coding dimension for speech-in-noise perception.