Inaudible attack has brought growing concerns over security of voice assistants. With a well-designed inaudible signal, an adversary can force the voice assistant to execute commands inaudibly like "Siri, open the door". It is challenging to defend against ultrasonic attacks without modifying the hardware. In this paper, we proposed a light-weight system named IMUSHIELD to defend voice assistant against inaudible attack. By comparing the different response of signal from microphone and inertial measurement units (IMUs) to different frequencies on smartphones. IMUSHIELD is able to detect the attacks without modifying the hardware. We have prototyped our method on a number of smartphones and test the performance of IMUSHIELD comprehensively in the real world, the result shows that our average detection accuracy exceeded 90 %.