Prevalence rates of pediatric obesity continue to rise worldwide. Adipose tissue (AT) development and expansion initiate in the fetus and extend throughout the lifespan. This paper presents an overview of the AT developmental trajectories from the intrauterine period to adolescence; factors determining adiposity expansion are also discussed. The greatest fetal increases in AT were observed in the third pregnancy trimester, with growing evidence suggesting that maternal health and nutrition, toxin exposure, and genetic defects impact AT development. From birth up to six months, healthy term newborns experience steep increases in AT; but a subsequent reduction in AT is observed during infancy. Important determinants of AT in infancy identified in this review included feeding practices and factors shaping the gut microbiome. Low AT accrual rates are maintained up to puberty onset, at which time, the pattern of adiposity expansion becomes sex dependent. As girls experience rapid increases and boys experience decreases in AT, sexual dimorphism in hormone secretion can be considered the main contributor for changes. Eating patterns/behaviors and interactions between dietary components, gut microbiome, and immune cells also influence AT expansion. Despite the plasticity of this tissue, substantial evidence supports that adiposity at birth and infancy highly influences its levels across subsequent life stages. Thus, a unique window of opportunity for the prevention and/or slowing down of the predisposition toward obesity, exists from pregnancy through childhood.