An evaluation system for postgraduate pediatric residency programs: report of a 3-year experience

Abstract

The way a postgraduate medical training program is organized and the capacity of faculty members to function as tutors and to organize effective professional experiences are among the elements that affect the quality of training. An evaluation system designed to target these elements has been implemented within the framework of the Pediatric Residency Program of the University of Padua (Italy). The aim of this report is to describe some aspects of the experience gained in the first 3 years of implementation of the system (2013–2015). Data were collected using four validated questionnaires: the “Resident Assessment Questionnaire”, the “Tutor-Assessment Questionnaire”, the “Rotation-Assessment Questionnaire”, and the “Resident Affairs Committee-Assessment Questionnaire”. The response rate was 72% for the “Resident Assessment Questionnaires”; 78% for the “Tutor-/Rotation-Assessment Questionnaires” and 84% for the “Resident Affair Committee-Assessment Questionnaires”. The scores collected were validated by psychometric tests. Conclusion: The high rates of completed questionnaires returned and the psychometric validation of the results collected indicate that the evaluation system reported herein can be effectively implemented. Efforts should be made to refine this system and, more importantly, to document its impact in improving the Pediatric Residency Program. What is known: • The elements that influence the quality of postgraduate training programs and the knowledge, performance, and competences of residents must be regularly assessed. • Comprehensive evaluation systems for postgraduate residency programs are not universally implemented also because quite often common guidelines and rules, well-equipped infrastructures, and financial resources are missing. What is new: • We show the feasibility of implementing an evaluation system that targets some of the key elements of a postgraduate medical training program in Italy, a European country in which the regulations governing training programs and, notably, the evaluation of residents are still being developed.

An Item Response Theory-Based Scoring of the South Oaks Gambling Screen–Revised Adolescents

“…There are several applications of IRT to the development and validation of measurement instruments (e.g., Anselmi et al, 2011, 2015; Anselmi, Vianello, & Robusto, 2013; Anselmi, Vianello, Voci, et al, 2013; C. Brown et al, 2015; Childs et al, 2000; Colledani, Anselmi, et al, 2018; Colledani et al, 2019a, 2019b; Colledani, Robusto, et al, 2018; Da Dalt et al, 2013, 2015, 2017; Rossi Ferrario et al, 2019; Rouse et al, 1999; Sotgiu et al, 2019; Vidotto et al, 2018; Zanon et al, 2016). IRT has been used to investigate the psychometric properties of different instruments for measuring problem gambling.…”

Literature review: influential factors on graduate professor’s performance

“…A Quality Guarantee Model for Graduate Programs was created to optimize processes of autoregulation, self-assessment and accreditation of master and doctorate degree programs of the Catholic University of Santísima Concepcíon, in Chile (Butter, Aguilera, Quintana, Pérez, & Valenzuela, 2017). In Italy, a graduate assessment system was designed and implemented in the Pediatric Residency Program of the University of Padova, in which the study describes some experience aspects acquired in the former 3 years of the system implementation (Da Dalt et al, 2017). In Spain an instrument also is drawn, based in the use of a factor analysis technique to measure educational quality in a graduate course (Moguerza, Fernández-Muñoz, Redchuk, Cardone-Riportella, & Navarro-Pardo, 2017).…”

A Comparison of Classical and Modern Measures of Internal Consistency

“…Less known than KR20 and α, R develops within modern test theory and, more precisely, within Rasch models. There are several applications of these models to the development and validation of measurement instruments (see, e.g., Duncan et al, 2003; Cole et al, 2004; Vidotto et al, 2006, 2007, 2010; Pallant and Tennant, 2007; Shea et al, 2009; Anselmi et al, 2011, 2013a,b, 2015; Da Dalt et al, 2013, 2015, 2017; Balsamo et al, 2014; Liu et al, 2017; Rossi Ferrario et al, 2019; Sotgiu et al, 2019). Rasch models characterize the responses of persons to items as a function of person and item measures (in the Rasch framework, the terms “person measure” and “item measure” are used to denote the values of the person parameter and item parameter, respectively).…”