Pleomorphic xanthoastrocytomas (PXA) are rare, accounting for < 1% of all astrocytomas. Literature on the clinical course and treatment outcomes of PXAs is limited. The study aimed to determine prognosis and treatment strategies for PXAs. Patients who had PXAs surgery between 2000–2021 were retrospectively analyzed for demographics and radiological characteristics. Initial and salvage treatment outcomes were recorded. Overall, 40 and 9 patients had grade 2 and 3 PXAs; their 5-year progression-free survival (PFS) rates were 75.8% and 37.0%, respectively (p = 0.003). Univariate analysis revealed that strong T1 enhancement (p = 0.036), infiltrative tumor margins (p < 0.001), peritumoral edema (p = 0.003), WHO grade (p = 0.005), and gross total resection (p = 0.005) affected the PFS. Multivariate analysis revealed that the WHO grade (p = 0.010) and infiltrative tumor margins (p = 0.008) influenced the PFS. The WHO grade (p = 0.027) and infiltrative tumor margins (p = 0.027) also affected the overall survival (OS). Subgroup analysis for grade 2 PXAs revealed no significant associations between adjuvant radiation therapy and the PFS and OS. This study highlighted the heterogeneous nature of PXAs and its impact on patient prognosis. Infiltrative tumor margins emerged as a key prognostic factor. Our findings have emphasized the prognostic relevance of radiological features and the need for larger studies on comprehensive management.