In this work, a series of the thermoplastic elastomer (TPE) composites filled with micro CaCO3 and hollow glass microspheres (HGM) via alloying process were investigated. The plastic phase of TPE composite referred to polypropylene (PP) and the rubber phase was chosen by styrene butadiene styrene (SBS). SEM observation conformed that the inorganic particles were mainly dispersed in continuous PP phase and SBS phase remained in an isolated structure. The sound insulation property was measured by a four-microphone impedance tube. Compared to pure polymer sample, both CaCO3 and HGM-filled TPE composites exhibited greatly enhanced soundproof efficiency, which increased the sound transmission loss value from original 29 dB to 45 dB. The soundproof mechanism was investigated in detail. The damping of acoustic energy was contributed by the extended difference between the stiffness and acoustic impedance between plastic phase and rubber phase. The dispersion and property of inorganic fillers were considered as key factors to increase the dissipation of sound wave. Meanwhile, the mechanical properties of TPE composites were enhanced due to the addition of inorganic fillers.