BackgroundTeaching clinical audit skills to nascent health professionals is one strategy to improve frontline care. The undergraduate medical curriculum at the University of Auckland provides improvement science theory and skills in Year 5 teaching, and the opportunity to put this into practice during an Obstetrics and Gynaecology (O&G) clinical attachment in Year 6. In 2015, a revised medical school curriculum at the university resulted in a planned reduction of the O&G attachment from five weeks to four, necessitating revision of the Year 6 Quality Improvement (QI) project. The aim of this study was to evaluate if the revised programme provided an important experiential learning opportunity for medical students without imposing an unsustainable burden on clinical services.MethodsBased on a CIPP (Context/Input/Process/Product) evaluation model, the study was conducted in several stages to get a sense of the context as the new programme was being planned (Context evaluation), the feasibility of an alternative approach to meet the educational need (Input evaluation), the implementation of the revised programme (Process evaluation) and finally, the programme outcomes (Product evaluation). We used multiple data sources (supervisors, students, academic administrators, and hospital staff) and data collection methods (questionnaires, focus groups, individual interviews, consultative workshops, student reports and oral presentations).ResultsThe context evaluation revealed the Year 6 QI programme to be valuable and contributed to O&G service improvements, however, the following concerns were identified: time to complete the project, timely topic selection and access to data, recognition of student achievement, and staff workload. The evaluation of the revised QI project indicated improvement in student perceptions of their QI knowledge and skills, and most areas previously identified as challenging, despite the concurrent reduction in the duration of the O&G attachment.ConclusionsApplying the CIPP model for evaluation to our revised QI programme enabled streamlining of procedures to achieve greater efficiency without compromising the quality of the learning experience, or increasing pressure on staff. A four week clinical rotation is adequate for medical educators to consider opportunities for including QI projects as part of student experiential learning.Electronic supplementary materialThe online version of this article (doi:10.1186/s12909-017-0927-y) contains supplementary material, which is available to authorized users.