Introduction Phyllodes tumors (PTs) are uncommon fibroepithelial breast tumors that occur in middle-aged women, and they tend to vary in biologic behavior. Surgical management is the standard therapy for the condition, but factors associated with recurrence remain unclear. The aim of this study was to evaluate clinical and surgical characteristics related to PT recurrences. Methods This retrospective cohort study included patients in southern Colombia who were diagnosed with PT and managed at a level I teaching and referral hospital over a nine-year period. Factors associated with recurrence were determined by Cox regression analysis. Results This study included 61 patients; their median age was 46 years [interquartile range (IQR): 39-55 years]. Pathologically, 37 tumors (60.7%) were classified as low-grade. The median tumor size was 7 cm (IQR: 4-11.5 cm). Thirty-nine (63.9%) patients underwent quadrantectomy. Nine patients (14.8%) experienced tumor recurrence, with the median time to recurrence being one year (IQR: 0.5-2 years). Distant metastasis was observed in four patients (6.6%) at a median of nine months (IQR: 0.4-2.5 years). Univariate analyses showed that patients with high-grade tumors [hazard ratio (HR): 2.90, p = 0.148] and those who underwent mastectomy (HR: 2.90, p = 0.460) were at higher risk of recurrence. Conclusion PT recurrence may be associated with biological features, the extent of local excision, tumor size, and negative margins. However, multicenter data are needed to confirm these findings.