The COVID-19 pandemic represents a worldwide threat to mental health. To optimize the allocation of health care resources, research on specific vulnerability factors, such as health anxiety, intolerance of uncertainty, and distress (in)tolerance, and particularly their effect on the time course of SARS-CoV-2 related anxiety appears crucial for supporting high risk groups suffering from elevated mental distress during the pandemic. N = 887 participants (78.4% female; Mage = 38.15, SD = 17.04) completed an online survey in Germany (April to mid-May 2020), comprising measures of SARS-CoV-2 related anxiety, health anxiety, safety and preventive behavior, intolerance of uncertainty, and distress intolerance. Higher levels of health anxiety pre and during COVID-19 were associated with an initially intensified increase (b = 1.10, p < 0.001), but later on a more rapid dampening (b = −0.18, p < 0.001) of SARS-CoV-2 related anxiety. SARS-CoV-2 related preventive behavior was intensified by both pre (b = 0.06, p = 0.01) and during (b = 0.15, p < 0.001) COVID-19 health anxiety, while reassurance behavior only was associated with health anxiety during COVID-19 (b = 0.14, p < 0.001). Distress intolerance and intolerance of uncertainty did not moderate the relationship between health anxiety and SARS-CoV-2 related anxiety and behavior. The results suggest detrimental effects of health anxiety on the emotional and behavioral response to virus outbreaks.