Phenols are a wide family of phytochemicals that are characterized by large chemical diversity and are considered to bioactive molecules of foods, beverages, and botanicals. Although they have a multitude of biological actions, their beneficial effects are rarely evidenced in clinical practice. This may occur due to the presence of numerous confounders, such as the modulation of phenol bioavailability, which can be regulated by microbiota, age, sex-gender. Sex-gender is an important determinant of health and well-being, and has an impact on environmental and occupational risks, access to health care, disease prevalence, and treatment outcomes. In addition, xenobiotic responses may be strongly influenced by sex-gender. This review describes how sex–gender differentially influences the activities of phenols also in some critical periods of women life such as pregnancy and lactation, considering also the sex of fetuses and infants. Thus, sex–gender is a variable that must be carefully considered and should be used to propose directions for future research on the road to tailored medicine and nutrition.