Musical compositions are distinguished by their unique rhythmic patterns, determined by subtle differences in how regular beats are subdivided. Precise perception of these subdivisions is essential for discerning nuances in rhythmic patterns. While musical rhythm typically comprises sound elements with a variety of timbres or spectral cues, the impact of such spectral variation on our perception of rhythmic patterns remains unclear. Here, we show that consistency in spectral cue affects perceptual accuracy in discriminating subdivided rhythmic patterns. We conducted online experiments using rhythmic sound sequences consisted of band-passed noise bursts to measure discrimination accuracy. Participants were asked to discriminate between a swing-like rhythm sequence, characterized by a 2:1 in interval ratio, and its more or less exaggerated version. This task was also performed under two additional rhythm ratio conditions: an inversed-swing rhythm (1:2 ratio) and a regular subdivision (1:1 ratio). The center frequency of band noises was either held constant or alternated between two values. Our results revealed a significant decrease in discrimination accuracy when the center frequency was alternated, irrespective of the rhythm ratio condition. This suggests that rhythm perception is not only shaped by temporal structure but also affected by spectral properties.