Eating meals as a family is associated with multiple positive nutritional and emotional outcomes for parents and children. Although the benefits of mealtimes extend to all families, families of color and those in poverty face disproportional barriers to eating frequent meals together. No previous study has properly attended to the heterogeneity of racial and ethnic groups in the United States when assessing mealtime barriers. Focusing on adverse childhood experiences (ACEs), our analyses tested whether an increased number of adversities, and particular types of adversities, were associated with a decrease in mealtime frequency for different racial/ethnic families in the United States. Utilizing a large, nationally representative sample of families (n = 59,963), results showed that higher cumulative ACE scores reduced the number of days a family eats meals together in a given week. For specific ACE items, seven out of nine individual ACE items were associated with decreased mealtimes (excluding parental death and racial discrimination). Stratification resulted in varied associations between accumulated and individual ACE items and mealtime frequency depending on racial/ethnic group. Asian families in particular had greater odds of infrequent meals than other families, while Native American, Alaskan Native, Pacific Islander, and other racial/ethnic families were unaffected by increased ACEs. Results indicate that a family’s accumulation of multiple adversities may impede mealtimes by either forcing families to prioritize the management of other stressors or by depleting the physical and mental resources needed to establish a routine. Focusing on family cultural traditions as protective factors may be an area of future intervention.