As the data concerning element concentrations in human milk (HM) samples and their intake by infants are lacking in Poland, the present study aimed to explore this issue. The material consisted of HM samples obtained from 30 exclusively breastfeeding mothers during 4–6 weeks postpartum. Additionally, to identify the factors that may potentially affect HM composition, information regarding maternal data (anthropometry, body composition, and diet) was also collected. Maternal diet was assessed with two methods—a food frequency questionnaire and 3-day dietary records. In total, 18 essential and non-essential elements were determined. For the elements analysis, we used inductively coupled plasma quadrupole mass spectrometry. Most of the elements (n = 11, 61%) were detected in all HM samples. In all HM samples tin concentration was higher (5.67 ± 2.39 μg/L) than the usual range reported by the World Health Organization (~ 1.0 μg/L). HM cadmium content was positively associated with maternal salty snacks intake (r = 0.502, p = 0.005), arsenic with whole-grain products intake (r = 0.37, p = 0.043), and mercury concentration with fruits and seeds/nuts consumption (r = 0.424, p = 0.042 and r = 0.378, p = 0.039, respectively). Higher HM lead concentration was predicted by maternal age (95% CI [0.94–0.97]), intake of fish (95% CI [1.01–1.03]), and vegetables (95% CI [1.02–1.06]). The highest infants’ intake was observed for copper (35.24 ± 12.48) and the lowest for arsenic (0.076 ± 0.102). Infants’ exposure to lead was associated with maternal frequency consumption of canned fish (p = 0.0045). There is a need to perform further research on this topic to maximize the benefits of breastfeeding by minimizing maternal and infant exposure to potentially toxic elements.