Although several studies have applied single-cell approaches to explore gene expression changes in aged brains, they were limited by the relatively shallow sampling of brain cell populations, and thus may have failed to capture aspects of the molecular signatures and dynamics of rare cell types associated with aging and diseases. Here, we set out to investigate the age-dependent dynamics of transcription and chromatin accessibility across diverse brain cell types. With EasySci, an extensively improved single-cell combinatorial indexing strategy, we profiled ~1.5 million single-cell transcriptomes and ~400,000 single-cell chromatin accessibility profiles across mouse brains spanning different ages, genotypes, and both sexes. With a novel computational framework designed for characterizing cellular subtypes based on the expression of both genes and exons, we identified > 300 cell subtypes and deciphered the underlying molecular programs and spatial locations of rare cell types (e.g., pinealocytes, tanycytes) and subtypes. Leveraging these data, we generate a global readout of age-dependent cell population dynamics with high cellular subtype resolution, providing insights into cell types that expand (e.g., rare astrocytes and vascular leptomeningeal cells in the olfactory bulb, reactive microglia and oligodendrocytes) or are depleted (e.g., neuronal progenitors, neuroblasts, committed oligodendrocyte precursors) as age progresses. Furthermore, we explored cell-type-specific responses to genetic perturbations associated with Alzheimer's disease (AD) and identify rare cell types depleted (e.g., mt-Cytb+, mt-Rnr2+ choroid plexus epithelial cells) or enriched (e.g., Col25a1+, Ndrg1+ interbrain and midbrain neurons) in both AD models. Key findings are consistent between males and females, validated across the transcriptome, chromatin accessibility, and spatial analyses. Finally, we profiled a total of 118,240 single-nuclei transcriptomes from twenty-four human brain samples derived from control and AD patients, revealing highly cell-type-specific and region-specific gene expression changes associated with AD pathogenesis. Critical AD-associated gene signatures were validated in both human and mice. In summary, these data comprise a rich resource for exploring cell-type-specific dynamics and the underlying molecular mechanisms in both normal and pathological mammalian aging.