Background: Physicians play a crucial frontline role in the COVID-19 pandemic, which may involve high levels of anxiety. We aimed to investigate the association between pandemic-related stress factors (PRSF) and anxiety and to evaluate the potential effect of resilience on anxiety among physicians. Methods: A self-report digital survey was completed by 1106 Israeli physicians (564 males and 542 females) during the COVID-19 outbreak. Anxiety was measured by the 8-item version of the Patient-Reported Outcomes Measurement Information System. Resilience was evaluated by the 10-item Connor-Davidson Resilience Scale. Stress was assessed using a PRSF inventory. Results: Physicians reported high levels of anxiety with a mean score of 59.20 ± 7.95. We found an inverse association between resilience and anxiety. Four salient PRSF (mental exhaustion, anxiety about being infected, anxiety infecting family members, and sleep difficulties) positively associated with anxiety scores. Conclusions: Our study identified specific PRSF including workload burden and fear of infection that are associated with increased anxiety and resilience that is associated with reduced anxiety among physicians.