Both food swamps and food deserts have been associated with racial, ethnic, and socioeconomic disparities in obesity rates. Little is known about how the distribution of food deserts and food swamps relate to disparities in self-reported dietary habits, and health status, particularly for historically marginalized groups. In a national U.S. sample of 4305 online survey participants (age 18+), multinomial logistic regression analyses were used to assess by race and ethnicity the likelihood of living in a food swamp or food desert area. Predicted probabilities of self-reported dietary habits, health status, and weight status were calculated using the fitted values from ordinal or multinomial logistic regression models adjusted for relevant covariates. Results showed that non-Hispanic, Black participants (N = 954) were most likely to report living in a food swamp. In the full and White subsamples (N = 2912), the perception of residing in a food swamp/desert was associated with less-healthful self-reported dietary habits overall. For non-Hispanic Blacks, regression results also showed that residents of perceived food swamp areas (OR = 0.66, p < 0.01, 95% CI (0.51, 0.86)) had a lower diet quality than those not lving in a food swamp/food desert area. Black communities in particular may be at risk for environment-linked diet-related health inequities. These findings suggest that an individual’s perceptions of food swamp and food desert exposure may be related to diet habits among adults.