Background Up to one-third of the patients suffering from eosinophilic esophagitis (EoE) present a refractory form, as defined by nonresponsiveness in clinical, endoscopic, or histological assessment after first-line therapy. Several studies recently investigated which factors can influence the development of this disease, but very few analyzed the factors underlying refractory EoE. Methods Medical charts of patients affected by EoE were retrospectively evaluated. Phenotyping of patients was conducted according to demographic, clinical, histological, and treatment variables. Then, patients were divided into responder and nonresponder to therapy and distinguished among children and adults. Results Forty-five children and 35 adult EoE patients were included. In the pediatric population, female sex (p < 0.05) and a higher score of visual analogue scale (VAS) at the follow-up visit (p = 0.02) were significantly associated to the risk of refractory EoE. Among adults, statistical significance was reached for years of follow-up (p = 0.001), diagnostic delay (p = 0.03), use of antibiotics during infancy (p = 0.01), and food allergy (p = 0.04). Conclusions Our study highlighted female sex and a higher VAS score at the time of follow-up visits as risk factors for refractory EoE in children, while the risk factors in adults were identified as fewer years of follow-up, greater diagnostic delay, use of antibiotics during infancy, and food allergy.