Here, we mapped cell-type specific chromatin domain organization in adult mouse cerebral cortex and report strong enrichment of Endogenous Retrovirus 2 (ERV2) repeat sequences in the neuron-specific heterochromatic 'B2NeuN+' megabase-scaling subcompartment. Comparative chromosomal conformation mapping in Mus spretus and Mus musculus revealed neuron-specific reconfigurations tracking recent ERV2 retrotransposon expansions in the murine germline, with significantly higher B2 megadomain contact frequencies at sites with ongoing ERV2 insertions in Mus musculus. Ablation of the retrotransposon silencer Kmt1e/Setdb1 triggered B2 megadomain disintegration and rewiring with open chromatin domains enriched for cellular stress response genes, along with severe neuroinflammation and proviral assembly of ERV2/Intracisternal-A-Particles (IAPs) infiltrating dendrites and spines. We conclude that neuronal megadomain architectures include evolutionarily adaptive heterochromatic organization which, upon perturbation, unleashes ERV proviruses with strong tropism within mature neurons.