The association between depressive symptoms and vegetarian diets is controversial. This study examines the cross-sectional association between depressive symptoms and vegetarian diets while controlling for potential confounders. Among 90,380 subjects from the population-based Constances cohort, depressive symptoms were defined by a score ≥19 on the Centre of Epidemiologic Studies-Depression (CES-D) scale and diet types (omnivorous, pesco-vegetarian, lacto-ovo-vegetarian and vegan) were determined with a food frequency questionnaire. Associations between depressive symptoms and diet were estimated through logistic regressions adjusting for socio-demographics, other foods, alcohol and tobacco consumption, physical activity and health-related concerns; specificity analyses considered the exclusion of any other food group. Depressive symptoms were associated with pesco-vegetarian and lacto-ovo-vegetarian diets in multivariable analyses (Odds-Ratio [95% confidence interval]: 1.43 [1.19–1.72] and 1.36 [1.09–1.70], respectively), especially in case of low legumes intake (p for interaction < 0.0001), as well as with the exclusion of any food group (e.g., 1.37 [1.24–1.52], 1.40 [1.31–1.50], 1.71 [1.49–1.97] for meat, fish and vegetables exclusion, respectively). Regardless of food type, the Odds-Ratio of depressive symptoms gradually increased with the number of excluded food groups (p for trend < 0.0001). Depressive symptoms are associated with the exclusion of any food group from the diet, including but not restricted to animal products.