Environmental influences such as perinatal stress have been shown to program the developing organism to adapt brain and behavioral functions to cope with daily life challenges. Evidence is now accumulating that the specific and individual effects of early life adversity on the functional development of brain and behavior emerge as a function of the type, intensity, timing and the duration of the adverse environment, and that early life stress (ELS) is a major risk factor for developing behavioral dysfunctions and mental disorders. Results from clinical as well as experimental studies in animal models support the hypothesis that ELS can induce functional “scars” in prefrontal and limbic brain areas, regions that are essential for emotional control, learning and memory functions. On the other hand, the concept of “stress inoculation” is emerging from more recent research, which revealed positive functional adaptations in response to ELS resulting in resilience against stress and other adversities later in life. Moreover, recent studies indicate that early life experiences and the resulting behavioral consequences can be transmitted to the next generation, leading to a transgenerational cycle of adverse or positive adaptations of brain function and behavior. In this review we propose a unifying view of stress vulnerability and resilience by connecting genetic predisposition and programming sensitivity to the context of experience-expectancy and transgenerational epigenetic traits. The adaptive maturation of stress responsive neural and endocrine systems requires environmental challenges to optimize their functions. Repeated environmental challenges can be viewed within the framework of the match/mismatch hypothesis, the outcome, psychopathology or resilience, depends on the respective predisposition and on the context later in life.