Physiological brain aging is characterized by gradual, substantial changes in cognitive ability, accompanied by chronic activation of the neural immune system. This form of inflammation, termed inflammaging, in the central nervous system is primarily enacted through microglia, the resident immune cells. The endocannabinoid system, and particularly the cannabinoid receptor 2 (CB2R), is a major regulator of the activity of microglia and is upregulated under inflammatory conditions. Here, we elucidated the role of the CB2R in physiological brain aging. We used CB2R−/− mice of progressive ages in a behavioral test battery to assess social and spatial learning and memory. This was followed by detailed immunohistochemical analysis of microglial activity and morphology, and of the expression of pro-inflammatory cytokines in the hippocampus. CB2R deletion decreased social memory in young mice, but did not affect spatial memory. In fact, old CB2R−/− mice had a slightly improved social memory, whereas in WT mice we detected an age-related cognitive decline. On a cellular level, CB2R deletion increased lipofuscin accumulation in microglia, but not in neurons. CB2R−/− microglia showed an increase of activity markers Iba1 and CD68, and minor upregulation in tnfa and il6 expression and downregulation of ccl2 with age. This was accompanied by a change in morphology as CB2R−/− microglia had smaller somas and lower polarity, with increased branching, cell volume, and tree length. We present that CB2Rs are involved in cognition and age-induced microglial activity, but may also be important for microglial activation itself.