Ultrasound-responsive polymeric materials have received a tremendous amount of attention from scientists for several decades. Compared to other stimuli-responsive materials (such as UV-, thermal-, and pH-responsive materials), these smart materials are more applicable since they allow more efficient drug delivery and targeted treatment by fairly non-invasive means. This review describes the recent advances of such ultrasound-responsive polymer-based drug delivery systems and illustrates various applications. More specifically, the mechanism of ultrasound-induced drug delivery, typical formulations, and biomedical applications (tumor therapy, disruption of blood-brain barrier, fighting infectious diseases, transdermal drug delivery, and enhanced thrombolysis) are summarized. Finally, a perspective on the future research directions for the development of ultrasound-responsive polymeric materials to facilitate a clinical translation is given.