ObjectiveNon-medical knowledge-based sub-competencies (multitasking, professionalism, accountability, patient-centered communication, and team management) are challenging for a supervising emergency medicine (EM) physician to evaluate in real-time on shift while also managing a busy emergency department (ED). This study examines residents’ perceptions of having a medical education specialist shadow and evaluate their nonmedical knowledge skills.MethodsMedical education specialists shadowed postgraduate year 1 and postgraduate year 2 EM residents during an ED shift once per academic year. In an attempt to increase meaningful feedback to the residents, these specialists evaluated resident performance in selected non-medical knowledge-based Accreditation Council of Graduate Medical Education (ACGME) sub-competencies and provided residents with direct, real-time feedback, followed by a written evaluation sent via email. Evaluations provided specific references to examples of behaviors observed during the shift and connected these back to ACGME competencies and milestones.ResultsTwelve residents participated in this shadow experience (six post graduate year 1 and six postgraduate year 2). Two residents emailed the medical education specialists ahead of the scheduled shadow shift requesting specific feedback. When queried, five residents voluntarily requested their feedback to be included in their formal biannual review. Residents received milestone scores and narrative feedback on the non-medical knowledge-based ACGME sub-competencies and indicated the shadow experience and subsequent feedback were valuable.ConclusionMedical education specialists who observe residents over the course of an entire shift and evaluate non-medical knowledge-based skills are perceived by EM residents to provide meaningful feedback and add valuable information for the biannual review process.