Sleep laboratory studies find that restricted sleep duration leads to worse short-term cognition, especially memory. Observational studies find associations between self-reported sleep duration or quality and cognitive function. However self-reported sleep characteristics might not be highly accurate, and misreporting could relate to cognition. In the Sleep Study of the National Social Life, Health, and Aging Project (NSHAP), a nationally representative cohort of older US adults (2010–2015), we examined whether self-reported and actigraph-measured sleep are associated with cross-sectional cognitive function and 5-year cognitive decline. Cognition was measured with the survey adaptation of the multidimensional Montreal Cognitive Assessment (MoCA-SA). At baseline (n = 759), average MoCA-SA score was 14.1 (standard deviation, 3.6) points of a possible 20. In cross-sectional models, actigraphic sleep-disruption measures (wake after sleep onset, fragmentation, percentage sleep, and wake bouts) were associated with worse cognition. Sleep disruption measures were standardized, and estimates of association were similar (range, −0.37 to −0.59 MoCA-SA point per standard deviation of disruption). Actigraphic sleep-disruption measures were also associated with odds of 5-year cognitive decline (4 or more points), with wake after sleep onset having the strongest association (odds ratio = 1.43, 95% confidence interval: 1.04, 1.98). Longitudinal associations were generally stronger for men than for women. Self-reported sleep showed little association with cognitive function.