The COVID-19 pandemic has had an emotional impact on healthcare professionals at different levels of care, and it is important to understand the levels of anxiety of hospital personnel (HP) compared to those of primary care personnel (PCP). The objectives herein were to assess the differences in anxiety levels between these populations and to detect factors that may influence them. The anxiety levels (measured using the Hospital Anxiety and Depression (HAD) scale) of the HP and PCP groups were compared using data collected from a cross-sectional study. The secondary variables included demographic and health data, confinement factors, contact with COVID-19 patients, having suffered from COVID-19, perceptions of protection, caregiver overload, threat, and satisfaction with management. We found anxiety “case” (35.6%) and “at-risk” (21%), with statistically significant differences in the group “at risk”, and higher scores in the PCP group. The factors associated with the perception of threat and protection were significant determinants of an increase in anxiety, with all of them showing statistically significant differences. There were greater symptoms of anxiety in the PCP group than the HP group (32% vs. 18%). The factors associated with the prevalence of anxiety symptoms were the perceptions of threat, protection, management, caregiver overload, and perceived degree of threat associated with COVID-19.