The composite rubber reinforced with hollow glass microsphere (HGM) was a promising composite material for noise reduction, and its sound insulation mechanism was studied based on an acoustic finite element simulation to gain the appropriate parameter with certain constraint conditions. The built simulation model included the air domain, polymer domain and inorganic particles domain. The sound insulation mechanism of the composite material was investigated through distributions of the sound pressure and sound pressure level. The influences of the parameters on the sound transmission loss (STL) were researched one by one, such as the densities of the composite rubber and HGM, the acoustic velocities in the polymer and inorganic particle, the frequency of the incident wave, the thickness of the sound insulator, and the diameter, volume ratio and hollow ratio of the HGM. The weighted STL with the 1/3 octave band was treated as the evaluation criterion to compare the sound insulation property with the various parameters. For the limited thicknesses of 1 mm, 2 mm, 3 mm and 4 mm, the corresponding optimal weighted STL of the composite material reached 14.02 dB, 19.88 dB, 22.838 dB and 25.27 dB with the selected parameters, which exhibited an excellent sound insulation performance and could promote the practical applications of the proposed composite rubber reinforced with HGM.