Human milk (HM) is dynamic and shows a high inter- and intra-individual variability. To characterize HM with precision, it is necessary to understand the factors that modulate its composition. The objective of this narrative review is to summarize the maternal, infant and methodological factors that affect HM composition. We searched SCOPUS and PubMed databases for articles related to factors that are known to or could potentially influence HM composition and volume across lactation periods. Our comprehensive review encompasses various maternal-, infant-related, and methodological factors that modulate aspects of HM composition including macro- and micronutrients, vitamins and minerals, as well as volume. The most profound changes were observed in HM lipids and lipophiles. Evidence exists for many of the infant-related factors known to affect the nutritive and non-nutritive components of HM (e.g., birth weight, gestational age, infant age/stage of lactation). In contrast, less is known with respect to maternal factors; where there is either limited research or conflicting evidence (e.g., maternal lifestyle, obstetric history, medical conditions), except for the mother's diet, for which there is a relatively well-established understanding. Equally, although many of the methodological factors (e.g., HM sampling, handling and analytics) are known to impact HM composition, few studies have investigated this as a primary outcome, making it an important area of future research in HM. Here we propose a systematic capture of numerous maternal- and infant-related characteristics to facilitate associative comparisons of HM data within and across studies. Additionally, it would be prudent to standardize the methodological aspects known to affect HM composition in analytics, not only for HM lipids and lipophiles, but also for those nutrients whose variability is yet less well-understood. Defining the factors determining HM composition with accuracy will open perspectives for maternal intervention to optimize milk composition for specific needs of infants.