Introduction: Learners may subconsciously change their behavior once they know they are being observed, and this Hawthorne effect should be considered when designing assessments of learner behavior. While there is a growing body of literature to suggest direct observation is the ideal standard for formative assessment, the best method to directly observe learners is unknown. We explored scheduled and unscheduled methods of direct observation among internal medicine residents in the outpatient continuity clinic to advance the understanding of both observation methods. Methods: We conducted a thematic analysis of faculty and internal medicine residents in an outpatient clinic setting. A semi-structured interview guide for focus group sessions was created. Focus groups were used to explore the internal medicine resident and core teaching faculty perceptions of the scheduled and unscheduled direct observation methods in the outpatient clinc. An experienced qualitative research interviewer external to the internal medicine residency was moderating the sessions. Eight peer focus groups were held. Abstraction of themes from focus group transcripts identified resident and faculty perceptions of the different observation methods. Results: Focus groups had 14 resident participants and 14 faculty participants. Unscheduled observations were felt to be more authentic than scheduled observations since residents perceived their behavior to be unmodified. Unscheduled observations allowed for increased numbers of observations per resident, which permitted more frequent formative assessments. Residents and faculty preferred remote video observation compared to in-room observation. Participants found direct observation a useful learning tool for high-yield, specific feedback. Conclusions: Unscheduled remote direct observation captures authentic clinical encounters while minimizing learner behavior modification. An unscheduled observation approach results in more frequent formative assessment and therefore in more instances of valuable feedback compared to scheduled observations. These findings can help guide the best practice approaches to direct clinical observation in order to enhance residents learning and experience.

REALLY GOOD STUFF both providers had to have masked telephone numbers for three-way conferences, which was addressed by using the online networking service Doximity for telephone calls and Zoom for video conferences. Finally, programme leaders worried that residents would respond poorly to an added responsibility given the overwhelming environment in and out of the hospital. This was addressed by implementing an opt-out option for residents who felt they could not participate for logistical or personal reasons. 3 | WHAT LE SSON S WERE LE ARNED? This transition yielded several lessons. First, we learned that it is possible to partner with administrators to reconcile educational and departmental financial priorities without compromising either-we stressed that threeway telemedicine visits would result in thorough evaluations with longer visit duration and therefore higher billing potential, justifying the administrative burden of restarting resident clinics. Second, we learned that residents are resilient, eager to learn and passionate about maintaining their patient relationships even in the midst of a pandemic-the majority of residents participated despite the opt-out option, and the alternative opt-in model would likely have resulted in low participation. Finally, we learned that the forced restructuring of a well-established workflow can reveal opportunities for improving education-several attending physicians remarked that they had rarely witnessed a resident take a complete history, arguably one of the most essential skills in neurological practice. 1 Previously inferring history-taking competence based on case presentations, they now directly evaluated residents' skills in interview efficiency such as asking progressively narrowing questions and communication skills such as responsive listening. After the pandemic, we plan to implement detailed history-taking evaluations and other elements of the new model to improve the quality of education in resident clinic.