We aimed to identify if maternal and infant factors were associated with neutral human milk oligosaccharides (HMOs) variability and examined the associations between HMOs concentration and infant growth and disease status in healthy Chinese mothers over a 6-month lactation period. We recruited mothers and their full-term infants as our subjects. At 1–5 days, 8–14 days, 4 weeks, and 6 months postpartum, all participants were interviewed to collect breast milk samples, obtain follow-up data and measure infant length and weight at their local hospital. A total of 23 neutral HMOs were analyzed by high performance liquid chromatography (HPLC)- mass spectrometer (MS). Secretor and Lewis phenotype were determined by the concentration of 2′-fucosyllactose (2′-FL) and Lacto-N-fucopentaose (LNFP)-II. The associations between maternal and infant factors with HMOs concentrations were investigated. A total of 464 human breast milk samples were collected from 116 mothers at four different time points. In total, 76.7% mothers were found to be Secretor and Lewis positive phenotype (Se+Le+), 17.2% were Se-Le+, 4.3% were Se+Le-, and 1.7% were Se-Le-. Several individual HMOs, including 2′-FL, Lactodifucotetraose (LDFT), LNFP-I were determined by Secretor phenotype. Most individual HMOs decreased at the later stage of lactation, except 3′-FL. We suggest that Secretor phenotype and lactation stage could influence most of the neutral HMOs. Concentrations of specific HMOs may be associated with maternal age, allergic history, pre-pregnancy body mass index (BMI), parity, delivery mode, infant gestational age and gender.