Purpose: To assess three advanced radiation therapy treatment planning tools on the intensity-modulated radiation therapy (IMRT) quality and consistency when compared to the clinically approved plans, referred as manual plans, which were planned without using any of these advanced planning tools.Materials and Methods: Three advanced radiation therapy treatment planning tools, including auto-planning, knowledge-based planning, and multiple criteria optimization, were assessed on 20 previously treated clinical cases. Three institutions participated in this study, each with expertise in one of these tools. The twenty cases were retrospectively selected from Cleveland Clinic, including five head-and-neck (HN) cases, five brain cases, five prostate with pelvic lymph nodes cases, and five spine cases. A set of general planning objectives and organs-at-risk (OAR) dose constraints for each disease site from Cleveland Clinic was shared with other two institutions. A total of 60 IMRT research plans (20 from each institution) were designed with the same beam configuration as in the respective manual plans. For each disease site, detailed isodoseline distributions and dose volume histograms for a randomly selected representative case were compared among the three research plans and manual plan. In addition, dosimetric endpoints of five cases for each site were compared.Results: Compared to the manual plans, the research plans using advanced tools showed substantial improvement for the HN patient cases, including the maximum dose to the spinal cord and brainstem and mean dose to the parotid glands. For the brain, prostate, and spine cases, the four types of plans were comparable based on dosimetric endpoint comparisons. Conclusion: With minimal planner interventions, advanced treatment planning tools are clinically useful, producing a plan quality similarly to or better than manual plans, improving plan consistency. For difficult cases such as HN cancer, advanced planning tools can further reduce radiation doses to numerous OARs while delivering adequate dose to the tumor targets. K E Y W O R D S auto-planning, knowledge-based planning, multiple criteria optimization ---