Previous studies have shown that music expertise relates to the hemispheric lateralization of music reading among musicians and non-musicians. However, it remains unclear that how music expertise modulates the hemispheric lateralization of music reading along the music learning trajectory and how music expertise modulates the hemispheric lateralization of reading different musical elements. This study examined how music expertise modulates the hemispheric lateralization of music reading in pitch elements (e.g., pitch, harmony), temporal elements (e.g., rhythm), and expressive elements (e.g., articulation) among musicians, music learners, and non-musicians. Musicians (n = 38), music learners (n = 26), and non-musicians (n = 33) worked on a set of divided visual field sequential matching tasks with four musical elements, i.e., pitch, harmony, rhythm, and articulation, in separate blocks. An eye-tracker was used to ensure participants' central fixation before each trial. Participants judged whether the first and second target stimuli were the same as quickly and accurately as possible. The findings showed that for musicians, no significant differences were observed between the left visual field (LVF) and the right visual field (RVF), suggesting musicians' bilateral representation in music reading. Music learners had an RVF/LH (left hemisphere) advantage over the LVF/RH (right hemisphere), suggesting music learners tended to be more left-lateralized in music reading. In contrast, non-musicians had an LVF/RH advantage over the RVF/LH, suggesting non-musicians tended to be more right-lateralized in music reading. In addition, music expertise correlates with the laterality index (LI) in music reading, suggesting that the better the overall performance in music expertise task, the greater the tendency to be more left-lateralized in music reading. Nonetheless, musicians, music learners, and non-musicians did not show different visual field effects in any individual musical elements respectively, suggesting the cognitive processes involved might share similar lateralization effects among the three groups when only one particular musical element is examined. In general, this study suggests the effect of music training on brain plasticity along the music learning trajectory. It also highlights the possibilities that bilateral or left hemispheric lateralization may serve as an expertise marker for musical reading.