Background/Objective: As dietary intake and endocrine metabolism are vastly different by sex, we evaluated differences in the association of diet quality with body composition between men and women. Subjects/Methods: Close to 2,000 participants from the Multiethnic Cohort completed calibrated quantitative food frequency questionnaires at cohort entry (1993–96) and clinic visit (2013–16), from which the Healthy Eating Index (HEI-2010) was computed. Adiposity measures were obtained through DXA and MRI at clinic visit. Multivariable-adjusted mean adiposity measures were estimated by tertiles of HEI-2010 scores using general linear regression. The associations of diet quality with high visceral fat (VAT) and non-alcoholic fatty liver disease (NAFLD) were examined by logistic regression. To assess sex differences, cross-product terms with HEI-2010 were added to the models. Results: Mean HEI-2010 scores were higher for women than men at cohort entry (67.4 vs. 64.0) and clinic visit (73.6 vs. 71.0). Past and current diet quality was inversely associated with adiposity measures in men and women. Although interaction terms were not significant, the magnitude of the slopes and differences in adjusted means across tertiles suggested a stronger association for women than men. When comparing individuals who maintained a high vs. poor quality diet over 20 years, women but not men showed significantly lower risks for high VAT, whereas high HEI-2010 scores predicted a lower risk of NAFLD in both sexes. Conclusions: The inverse association of diet quality with adiposity was similar in both sexes, but diet quality appeared to have a stronger influence on VAT in women than men.