Background: This study aimed to investigate the factors influencing the use of complementary and alternative medicines (CAMs) to manage stress during the COVID-19 pandemic in Taiwan. Methods: A cross-sectional survey was administered to community-dwelling adults between the ages of 46 and 75 years, and a total of 351 participants completed the questionnaire. Log-binominal regression analyses were fitted to explore the factors associated with the use of CAMs. Results: The mean age of the participants was 57.0 years, and 67.0% reported that they had used CAMs within the past three months. Middle-aged adults were more likely to use CAMs than late middle-aged adults and older adults (p < 0.001). Overall, the major CAMs utilized to relieve psychological stress were music therapies (37.6%), massage (31.1%), spinal manipulation (25.1%), relaxing therapies (24.2%), and reading scriptures or The Bible (23.9%). Religion and vegetarian diets were the most important factors influencing participants to use CAMs, especially music therapies, massage, and reading scriptures/The Bible. Conclusions: CAM use was very prevalent among middle-aged adults in Taiwan; in particular, music therapies were the most favored activities for reducing stress. Population-specific mental health interventions using music can be developed to improve stress management outcomes during public health emergencies.