Background Physicians are considered to be a high-risk population for a poor quality of life (QoL), but few studies of lifestyle factors include the QoL among them. Objectives This study aimed to investigate the relationship between lifestyle factors and a positive QoL among primary health care (PHC) physicians. Methods A cross-sectional study was conducted at 20 primary healthcare centers in Madinah, Saudi Arabia. A self-administered questionnaire was used, including sociodemographic characteristics, lifestyle data, and the short version of the World Health Organization Quality of Life questionnaire. Appropriate statistical analyses were used, including multivariate logistic regression models. Results The response rate was 85.7% (72/84) physicians. The mean score of the total QoL and its four studied domains (physical, psychological, social, and environmental) was relatively high, with no statistically significant difference between the consultants and general practitioners. The positive total QoL in this study was significantly lower among physicians with obesity (OR = 0.55, 95%CI = 0.25–0.97), those using butter and animal fat for cooking (OR = 0.10, 95%CI = 0.02–0.81), and those eating meals out > 3 times per week (OR = 0.30, 95%CI = 0.10–0.90). Although non-significant, vegetable consumption and a high level of physical activity were associated with a positive QoL, with adjusted ORs of 2.5 (95%CI = 0.82–7.58) and 1.5 (95%CI = 0.33–6.65), respectively. Conclusion The findings indicate a relatively good QoL among the participating physicians; however, a lower QoL was associated with unhealthy lifestyle factors. QoL was significantly associated with obesity, cooking practices, and eating meals from restaurants.