This study aimed to assess the content of caffeine and its metabolites—paraxanthine, theophylline, and theobromine—in breast milk according to selected factors. Samples of human milk were collected from 100 women living in the east–northeast region of Poland. Information on the consumption of beverages and foods containing caffeine was collected using a 3 day food record. The determination of caffeine and its metabolite content was performed using liquid chromatography–mass spectrometry (LC–MS/MS). This study research showed that more caffeine was found in the milk of women living in cities, with secondary education, aged 34–43, and also in milk from the 3rd and 4th lactation periods (p ≤ 0.05). Factors such as place of residence, level of education, age, and stage of lactation influenced the nutritional choices of breastfeeding women, which had an impact on the level of caffeine and its metabolites in breast milk. A positive correlation was found between the consumption of caffeine with food and drinks and its level in human milk.