As a powerful and effective analytical tool, surface-enhanced Raman scattering (SERS) has attracted considerable research interest in the fields of wearable flexible sensing and non-invasive point-of-care testing (POCT) medical diagnosis. In this mini-review, we briefly summarize the design strategy, the development progress of wearable SERS sensors and its applications in this field. We present SERS substrate analysis of material design requirements for wearable sensors and highlight the benefits of novel plasmonic particle-in-cavity (PIC)-based nanostructures for flexible SERS sensors, as well as the unique interfacial adhesion effect and excellent mechanical properties of natural silk fibroin (SF) derived from natural cocoons, indicating promising futures for applications in the field of flexible electronic, optical, and electrical sensors. Additionally, SERS wearable sensors have shown great potential in the fields of different disease markers as well as in the diagnosis testing for COVID-19. Finally, the current challenges in this field are pointed out, as well as the promising prospects of combining SERS wearable sensors with other portable health monitoring systems for POCT medical diagnosis in the future.