Neurons and oligodendrocytes are terminally differentiated cells that perform highly specialized functions, which depend on cascades of gene activation and repression to retain homeostatic control over a lifespan. Gene expression is regulated by threedimensional (3D) genome organisation, from local levels of chromatin compaction to the organisation of topological domains and chromosome compartments. Whereas our understanding of 3D genome architecture has vastly increased in the past decade, it remains difficult to study specialized cells in their native environment without disturbing their activity. To develop the application of Genome Architecture Mapping (GAM) in small numbers of specialized cells in complex tissues, we combined GAM with immunoselection. We applied immunoGAM to map the genome architecture of specific cell populations in the juvenile/adult mouse brain: dopaminergic neurons (DNs) from the midbrain, pyramidal glutamatergic neurons (PGNs) from the hippocampus, and oligodendrocyte lineage cells (OLGs) from the cortex. We integrate 3D genome organisation with single-cell transcriptomics data, and find specific chromatin structures that relate with cell-type specific patterns of gene expression. We discover abundant changes in compartment organisation, especially a strengthening of heterochromatin compartments which establish strong contacts spanning tens of megabases, especially in brain cells. These compartments contain olfactory and taste receptor genes, which are de-repressed in a subpopulation of PGNs with molecular signatures of long-term potentiation (LTP). We also show extensive reorganisation of topological domains where activation of neuronal or oligodendrocyte genes coincides with formation of new TAD borders.Finally, we discover loss of TAD organisation, or 'TAD melting', at long (>1Mb) neuronal genes when they are most highly expressed. Our work shows that the 3D 3 organisation of the genome is highly cell-type specific in terminally differentiated cells of the brain, and essential to better understand brain-specific mechanisms of gene regulation.