Background Diseases such as COVID-19 are spread through social contact. Reducing social contacts is required to stop disease spread in pandemics for which vaccines have not yet been developed. However, existing data on social contact patterns in the United States (U.S.) is limited. Method We use American Time Use Survey data from 2003–2018 to describe and quantify the age-pattern of disease-relevant social contacts. For within-household contacts, we construct age-structured contact duration matrices (who spends time with whom, by age). For both within-household and non-household contacts, we also estimate the mean number and duration of contact by location. We estimate and test for differences in the age-pattern of social contacts based on demographic, temporal, and spatial characteristics. Results The mean number and duration of social contacts vary by age. The biggest gender differences in the age-pattern of social contacts are at home and at work; the former appears to be driven by caretaking responsibilities. Non-Hispanic Blacks have a shorter duration of contact and fewer social contacts than non-Hispanic Whites. This difference is largely driven by fewer and shorter contacts at home. Pre-pandemic, non-Hispanic Blacks have shorter durations of work contacts. Their jobs are more likely to require close physical proximity, so their contacts are riskier than those of non-Hispanic Whites. Hispanics have the highest number of household contacts and are also more likely to work in jobs requiring close physical proximity than non-Hispanic Whites. With the exceptions of work and school contacts, the duration of social contact is higher on weekends than on weekdays. Seasonal differences in the total duration of social contacts are driven by school-aged respondents who have significantly shorter contacts during the summer months. Contact patterns did not differ by metro status. Age patterns of social contacts were similar across regions. Conclusion Social contact patterns differ by age, race and ethnicity, and gender. Other factors besides contact patterns may be driving seasonal variation in disease incidence if school-aged individuals are not an important source of transmission. Pre-pandemic, there were no spatial differences in social contacts, but this finding has likely changed during the pandemic.