Graphene, a two-dimensional nanomaterial, possess unique photoelectric properties that have potential application in designing optoelectronic devices. The tunable optical absorption is one of the most exciting properties that can be used to improve the performance of silicon modulators. However, the weak light–matter interaction caused by the size mismatch between the optical mode fields and graphene makes the graphene-on-silicon modulator (GOSM) has large footprint and high energy consumption, limiting the enhancement of modulation efficiency. Here, we propose a broadband GOSM with orthogonal hybrid plasmonic waveguides (HPWs) at near-infrared wavelengths. The orthogonal HPWs are designed to compress the interaction region of optical fields and enhance the light-graphene interaction. The results show that the GOSM has a modulation depth of 26.20 dB/μm, a footprint of 0.33 μm2, a 3 dB modulation bandwidth of 462.77 GHz, and energy consumption of 2.82 fJ/bit at 1.55 μm. Even working at a broad wavelength band ranging from 1.3 to 2 μm, the GOSM also has a modulation depth of over 8.58 dB/μm and energy consumption of below 4.97 fJ/bit. It is anticipated that with the excellent modulation performance, this GOSM may have great potential in broadband integrated modulators, on-chip optical communications and interconnects, etc.