Authenticity of instruction and student performance: a prospective randomised trial

Rochelle, Jeffrey S. La; Durning, Steven J.; Pangaro, Louis N.; Artino, Anthony R.; Vleuten, Cees van der; Schuwirth, Lambert

doi:10.1111/j.1365-2923.2011.03994.x

Cited by 42 publications

(44 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our study confirms that open-book type testing, which in our case involved searching the internet, costs time [19,23]. This is demonstrated by the fact that 30 minutes of internet access only allowed trainees to change an average of 8, predominantly factual, questions out of 150.…”

Section: Discussionsupporting

confidence: 80%

An exploratory study into the effect of time-restricted internet access on face-validity, construct validity and reliability of postgraduate knowledge progress testing

Dijksterhuis

Jozwiak

Braat³

et al. 2013

BMC Med Educ

View full text Add to dashboard Cite

BackgroundYearly formative knowledge testing (also known as progress testing) was shown to have a limited construct-validity and reliability in postgraduate medical education. One way to improve construct-validity and reliability is to improve the authenticity of a test. As easily accessible internet has become inseparably linked to daily clinical practice, we hypothesized that allowing internet access for a limited amount of time during the progress test would improve the perception of authenticity (face-validity) of the test, which would in turn improve the construct-validity and reliability of postgraduate progress testing.MethodsPostgraduate trainees taking the yearly knowledge progress test were asked to participate in a study where they could access the internet for 30 minutes at the end of a traditional pen and paper test. Before and after the test they were asked to complete a short questionnaire regarding the face-validity of the test.ResultsMean test scores increased significantly for all training years. Trainees indicated that the face-validity of the test improved with internet access and that they would like to continue to have internet access during future testing. Internet access did not improve the construct-validity or reliability of the test.ConclusionImproving the face-validity of postgraduate progress testing, by adding the possibility to search the internet for a limited amount of time, positively influences test performance and face-validity. However, it did not change the reliability or the construct-validity of the test.

show abstract

Section: Discussionsupporting

confidence: 80%

An exploratory study into the effect of time-restricted internet access on face-validity, construct validity and reliability of postgraduate knowledge progress testing

Dijksterhuis

Jozwiak

Braat³

et al. 2013

BMC Med Educ

View full text Add to dashboard Cite

show abstract

“…More specifically, the think-aloud protocol allowed us to explore if the selected contextual factors and our markers of cognitive load were associated clinical reasoning performance. Moreover, the videotapes provided a means to represent cases in a more authentic way than paper cases (La Rochelle et al 2011).…”

Section: Context Defined and Theoretic Frameworkmentioning

confidence: 99%

The impact of selected contextual factors on experts’ clinical reasoning performance (does context impact clinical reasoning performance in experts?)

Durning

Artino

Boulet

et al. 2011

Adv in Health Sci Educ

Self Cite

120

148

View full text Add to dashboard Cite

Context specificity, or the variation in a participant's performance from one case, or situation, to the next, is a recognized problem in medical education. However, studies have not explored the potential reasons for context specificity in experts using the lens of situated cognition and cognitive load theories (CLT). Using these theories, we explored the influence of selected contextual factors on clinical reasoning performance in internal medicine experts. We constructed and validated a series of videotapes portraying different chief complaints for three common diagnoses seen in internal medicine. Using the situated cognition framework, we modified selected contextual factors--patient, encounter, and/or physician--in each videotape. Following each videotape, participants completed a post-encounter form (PEF) and a think-aloud protocol. A survey estimating recent exposure from their practice to the correct videotape diagnoses was also completed. The time given to complete the PEF was randomly varied with each videotape. Qualitative utterances from the think-aloud procedure were converted to numeric measures of cognitive load. Survey and cognitive load measures were correlated with PEF performance. Pearson correlations were used to assess relations between the independent variables (cognitive load, survey of experience, contextual factors modified) and PEF performance. To further explore context specificity, analysis of covariance (ANCOVA) was used to assess differences in PEF scores, by diagnosis, after controlling for time. Low correlations between PEF sections, both across diagnoses and within each diagnosis, were observed (r values ranged from -.63 to .60). Limiting the time to complete the PEF impacted PEF performance (r = .2 to .4). Context specificity was further substantiated by demonstrating significant differences on most PEF section scores with a diagnosis (ANCOVA). Cognitive load measures were negatively correlated with PEF scores. The presence of selected contextual factors appeared to influence diagnostic more than therapeutic reasoning (r = -.2 to -.38). Contextual factors appear to impact expert physician performance. The impact observed is consistent with situated cognition and CLT's predictions. These findings have potential implications for educational theory and clinical practice.

show abstract

“…But the question is not whether the study is a good representation of the Breal world( it is not), but whether the constraints of the study methods invalidate the findings. Evidence from other studies 25 indicates that students learn clinical reasoning as well from written cases as from videos and from live standardized patients.…”

Section: Limitationsmentioning

confidence: 99%

Reflecting on Diagnostic Errors: Taking a Second Look is Not Enough

et al. 2015

View full text Add to dashboard Cite

BACKGROUND: An experimenter controlled form of reflection has been shown to improve the detection and correction of diagnostic errors in some situations; however, the benefits of participant-controlled reflection have not been assessed. OBJECTIVE: The goal of the current study is to examine how experience and a self-directed decision to reflect affect the accuracy of revised diagnoses. DESIGN: Medical residents diagnosed 16 medical cases (pass 1). Participants were then given the opportunity to reflect on each case and revise their diagnoses (pass 2). PARTICIPANTS: Forty-seven medical Residents in postgraduate year (PGY) 1, 2 and 3 were recruited from Hamilton Health Care Centres. MAIN MEASURES: Diagnoses were scored as 0 (incorrect), 1 (partially correct) and 2 (correct). Accuracies and response times in pass 1 were analyzed using an ANOVA with three factors-PGY, Decision to revise yes/no, and Case 1-16, averaged across residents. The extent to which additional reflection affected accuracy was examined by analyzing only those cases that were revised, using a repeated measures ANOVA, with pass 1 or 2 as a within subject factor, and PGY and Case or Resident as a between-subject factor. KEY RESULTS: The mean score at pass 1 for each level was PGY1, 1.17 (SE 0.50); PGY2, 1.35 (SE 0.67) and PGY3, 1.27 (SE 0.94). While there was a trend for increased accuracy with level, this did not achieve significance. The number of residents at each level who revised at least one diagnosis was 12/19 PGY1 (63 %), 9/11 PGY2 (82 %) and 8/17 PGY3 (47 %). Only 8 % of diagnoses were revised resulting in a small but significant increase in scores from Pass 1 to 2, from 1.20/2 to 1.22 /2 (t=2.15, p=0.03). CONCLUSIONS: Participants did engage in self-directed reflection for incorrect diagnoses; however, this strategy provided minimal benefits compared to knowing the correct answer. Education strategies should be directed at improving formal and experiential knowledge.

show abstract

Authenticity of instruction and student performance: a prospective randomised trial

Cited by 42 publications

References 17 publications

An exploratory study into the effect of time-restricted internet access on face-validity, construct validity and reliability of postgraduate knowledge progress testing

An exploratory study into the effect of time-restricted internet access on face-validity, construct validity and reliability of postgraduate knowledge progress testing

The impact of selected contextual factors on experts’ clinical reasoning performance (does context impact clinical reasoning performance in experts?)

Reflecting on Diagnostic Errors: Taking a Second Look is Not Enough

Contact Info

Product

Resources

About