Social behaviors, such as aggression or mating, proceed through a series of appetitive and consummatory phases1 that are associated with increasing levels of arousal2. How such escalation is encoded in the brain, and linked to behavioral action selection, remains an important unsolved problem in neuroscience. The ventrolateral subdivision of the murine ventromedial hypothalamus (VMHvl) contains neurons whose activity increases during male-male and male-female social encounters. Non-cell type-specific optogenetic activation of this region elicited attack behavior, but not mounting3. We have identified a subset of VMHvl neurons marked by the estrogen receptor 1 (Esr1), and investigated their role in male social behavior. Optogenetic manipulations indicated that Esr1+ (but not Esr1-) neurons are sufficient to initiate attack, and that their activity is continuously required during ongoing agonistic behavior. Surprisingly, weaker optogenetic activation of these neurons promoted mounting behavior, rather than attack, towards both males and females, as well as sniffing and close investigation (CI). Increasing photostimulation intensity could promote a transition from CI and mounting to attack, within a single social encounter. Importantly, time-resolved optogenetic inhibition experiments revealed requirements for Esr1+ neurons in both the appetitive (investigative) and the consummatory phases of social interactions. Combined optogenetic activation and calcium imaging experiments in vitro, as well as c-Fos analysis in vivo, indicated that increasing photostimulation intensity increases both the number of active neurons and the average level of activity per neuron. These data suggest that Esr1+ neurons in VMHvl control the progression of a social encounter from its appetitive through its consummatory phases, in a scalable manner that reflects the number or type of active neurons in the population.

SummaryAll animals possess a repertoire of innate (or instinctive1,2) behaviors, which can be performed without training. Whether such behaviors are mediated by anatomically distinct and/or genetically specified neural pathways remains a matter of debate3-5. Here we report that hypothalamic neural ensemble representations underlying innate social behaviors are shaped by social experience. Estrogen receptor 1-expressing (Esr1+) neurons in the ventrolateral subdivision of the ventromedial hypothalamus (VMHvl) control mating and fighting in rodents6-8. We used microendoscopy9 to image VMHvl Esr1+ neuronal activity in male mice engaged in these social behaviours. In sexually and socially experienced adult males, divergent and characteristic neural ensembles represented male vs. female conspecifics. But surprisingly, in inexperienced adult males, male and female intruders activated overlapping neuronal populations. Sex-specific ensembles gradually separated as the mice acquired social and sexual experience. In mice permitted to investigate but not mount or attack conspecifics, ensemble divergence did not occur. However, 30 min of sexual experience with a female was sufficient to promote both male vs. female ensemble separation and attack, measured 24 hr later. These observations uncover an unexpected social experience-dependent component to the formation of hypothalamic neural assemblies controlling innate social behaviors. More generally, they reveal plasticity and dynamic coding in an evolutionarily ancient deep subcortical structure that is traditionally viewed as a “hard-wired” system.

The amygdaloid complex is a group of nuclei that are thought to originate from multiple sites of the dorsal and ventral telencephalic neuroepithelium. The mechanisms that regulate their development are essentially unknown. We studied the role of Pax6 and Emx2, two transcription factors that regulate regional specification and growth of the telencephalon, in the morphogenesis of the amygdaloid complex. We used a set of specific marker genes that identify distinct amygdaloid nuclei to analyze Pax6/Small eye and Emx2 knock-out mutant mouse brains. We found that there is a selective requirement for Pax6, but not Emx2, in the formation a subset of nuclei within the amygdaloid complex. Specifically, structures that were not previously considered to be developmentally linked, the nucleus of the lateral olfactory tract and the lateral, basolateral, and basomedial nuclei, all appear to have a common requirement for Pax6. Together, our findings provide new insights into the origins and mechanisms underlying the development of the amygdaloid complex.

The amygdaloid complex consists of diverse nuclei that belong to distinct functional systems, yet many issues about its development are poorly understood. Here, we identify a stream of migrating cells that form specific amygdaloid nuclei in mice. In utero electroporation showed that this caudal amygdaloid stream (CAS) originated in a unique domain at the caudal telencephalic pole that is contiguous with the dorsal pallium, which was previously thought to generate only neocortical cells. The CAS and the neocortex share mechanisms for specification (transcription factors Tbr1, Lhx2 and Emx1/2) and migration (reelin and Cdk5). Reelin, a critical cue for migration in the neocortex, and Cdk5, which is specifically required for migration along radial glia in the neocortex, were both selectively required for the normal migration of the CAS, but not for that of other amygdaloid nuclei. This is first evidence of a dorsal pallial contribution to the amygdala, demonstrating a developmental and mechanistic link between the amygdala and the neocortex.

Defensive behaviors reflect underlying emotion states, such as fear. The hypothalamus plays a role in such behaviors, but prevailing textbook views depict it as an effector of upstream emotion centers, such as the amygdala, rather than as an emotion center itself. We used optogenetic manipulations to probe the function of a specific hypothalamic cell type that mediates innate defensive responses. These neurons are sufficient to drive multiple defensive actions, and required for defensive behaviors in diverse contexts. The behavioral consequences of activating these neurons, moreover, exhibit properties characteristic of emotion states in general, including scalability, (negative) valence, generalization and persistence. Importantly, these neurons can also condition learned defensive behavior, further refuting long-standing claims that the hypothalamus is unable to support emotional learning and therefore is not an emotion center. These data indicate that the hypothalamus plays an integral role to instantiate emotion states, and is not simply a passive effector of upstream emotion centers.DOI: http://dx.doi.org/10.7554/eLife.06633.001