Objective: Cross-sectional and long-term longitudinal studies have shown that engagement in diverse activities benefits cognitive performance in older age, but it is unknown whether the beneficial effect holds within persons on a daily basis. This study examines the within-person association between activity diversity and working memory on the same day and its time-lagged directionality between days. It also examines the effects of potential moderators on the within-person association, including age, education, processing speed, and crystallized intelligence, to understand who may benefit more from daily activity diversity. Method: We examined smartphone-based ambulatory assessment data from 150 community-dwelling older adults (aged 65–91 years) from Switzerland. Participants reported their present activity and completed a working memory task (i.e., numerical updating) seven times per day over 15 days. Activity diversity was calculated on a daily level and scores of working memory were averaged within a day. Age, education, processing speed, and crystallized intelligence were assessed in the laboratory at baseline. Results: Multilevel models showed that, within persons, higher daily activity diversity was positively associated with higher daily working memory. Moreover, the prior day’s greater activity diversity led to that day’s higher working memory, but not vice versa. There were no moderating effects of age, education, and crystallized intelligence, but partial evidence of a moderating effect of processing speed. Conclusions: Our results on within-person concurrent and time-lagged associations between daily activity diversity and daily working memory strengthen the existing evidence on the beneficial effect of activity diversity on older adults’ cognitive performance. Results are discussed in the context of cognitive reserve theory.