Breastfeeding (duration) can be positively associated with infant growth outcomes as well as improved cognitive functions during childhood and later life stages. (Prolonged) exposure to optimal lipid quantity and quality, i.e., the supramolecular structure of lipids, in mammalian milk, may contribute to these beneficial effects through nutritional early-life programming. In this pre-clinical study, we exposed male C57BL/6J mice from post-natal Days 16 to 42 (i.e., directly following normal lactation), to a diet with large lipid droplets coated with bovine milk fat globule membrane-derived phospholipids, which mimic more closely the supramolecular structure of lipid droplets in mammalian milk. We investigated whether exposure to this diet could affect growth and brain development-related parameters. As these outcomes are also known to be affected by the post-weaning social environment in mice, we included both individually housed and pair-wise housed animals and studied whether effects of diet were modulated by the social environment. After Day 42, all the animals were fed standard semi-synthetic rodent diet. Growth and body composition were assessed, and the mice were subjected to various behavioral tests. Individual housing attenuated adolescent growth, reduced femur length, and increased body fat mass. Adult social interest was increased due to individual housing, while cognitive and behavioral alterations as a result of different housing conditions were modest. The diet increased adolescent growth and femur length, increased lean body mass, reduced adolescent anxiety, and improved adult cognitive performance. These effects of diet exposure were comparable between individually and socially housed mice. Hence, early life exposure to a diet with lipid droplets that mimic the supramolecular structure of those in mammalian milk may improve adolescent growth and alters brain function in both socially and individually housed mice. These findings suggest that lipid structure in infant milk formula may be a relevant target for nutritional solutions, targeting both healthy infants and infants facing growth challenges.