Transforming MRCPsych theory examinations: digitisation and very short answer questions (VSAQs)

Scheeres, Karl; Agrawal, Niruj; Ewen, Stephanie; Hall, Ian

doi:10.1192/bjb.2021.23

Cited by 5 publications

(7 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Yet, MCQs are thought to be superior to TFQs because TFQs have a 50% chance of guessing the right answer (for MCQs, there is only a 20% chance) and offer little insight as to why a student answered correctly. Thus, TFQs are best suited to assessing surface-level medical knowledge ( Scheeres et al, 2022 ). As frontotemporal activity during the MCQs was higher than the TFQs, this observation further supports the postulation that MCQs are more stimulating than TFQs.…”

Section: Discussionmentioning

confidence: 99%

“…For undergraduates, Sam et al (2016) believed that reliance on assessment by single best answer MCQs (e.g., best out of 5 answers) could foster the learning of association and superficial understanding of medical knowledge in order to pass exams ( Sam et al, 2016 ). For postgraduate trainees or residents, Scheeres et al (2022) proposed the gradual introduction of SAQs to enable a more comprehensive assessment of their core medical knowledge and promote deeper learning strategies. A carefully phased introduction of alternative item formats, such as CSQs and SAQs, would enable a more robust assessment and provide opportunities for improving the assessment of theoretical knowledge ( Scheeres et al, 2022 ).…”

Section: Discussionmentioning

confidence: 99%

“…For postgraduate trainees or residents, Scheeres et al (2022) proposed the gradual introduction of SAQs to enable a more comprehensive assessment of their core medical knowledge and promote deeper learning strategies. A carefully phased introduction of alternative item formats, such as CSQs and SAQs, would enable a more robust assessment and provide opportunities for improving the assessment of theoretical knowledge ( Scheeres et al, 2022 ). The development of artificial intelligence or machine learning algorithms that can mark SAQs and CSQs would reduce manpower needs.…”

Section: Discussionmentioning

confidence: 99%

See 2 more Smart Citations

Functional near-infrared spectroscopy of medical students answering various item types

et al. 2023

View full text Add to dashboard Cite

BackgroundTraditionally, the effect of assessment item types including true/false questions (TFQs), multiple-choice questions (MCQs), short answer questions (SAQs), and case scenario questions (CSQs) is examined through psychometric qualities or student interviews. However, brain activity while answering such questions or items remains unknown. Functional near-infrared spectroscopy (fNIRS) can be used to safely measure cerebral cortex hemodynamic response during various tasks. Hence, this fNIRS study aimed to determine differences in frontotemporal cortex activity as medical students answered TFQs, MCQs, SAQs, and CSQs.MethodsIn total, 24 medical students (13 males and 11 females) were recruited in this study during their mid-psychiatry posting. Oxy-hemoglobin and deoxy-hemoglobin levels in the frontal and temporal regions were measured with a 52-channel fNIRS system. Participants answered 9–18 trials under each of the four types of tasks that were based on their psychiatry curriculum during fNIRS measurements. The area under the oxy-hemoglobin curve (AUC) for each participant and each item type was derived. Repeated measures ANOVA with post-hoc Bonferroni-corrected pairwise comparisons were used to determine differences in oxy-hemoglobin AUC between TFQs, MCQs, SAQs, and CSQs.ResultsOxy-hemoglobin AUC was highest during the CSQs, followed by SAQs, MCQs, and TFQs in both the frontal and temporal regions. Statistically significant differences between different types of items were observed in oxy-hemoglobin AUC of the frontal region (p ≤ 0.001). Oxy-hemoglobin AUC in the frontal region was significantly higher during the CSQs than TFQ (p = 0.005) and during the SAQ than TFQ (p = 0.025). Although the percentage of correct responses was significantly lower in MCQ than in the other item types, there was no correlation between the percentage of correct response and oxy-hemoglobin AUC in both regions for all four item types (p > 0.05).ConclusionCSQs and SAQs elicited greater hemodynamic response than MCQs and TFQs in the prefrontal cortex of medical students. This suggests that more cognitive skills may be required to answer CSQs and SAQs.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Functional near-infrared spectroscopy of medical students answering various item types

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Moreover, feedback suggested that VSAQ phrasing might not always have been clear enough. This led to uncertainty regarding the level of specificity of the desired answer and highlights the importance of a well-designed VSAQ in order to profit from their advantages over MCQs [8, 31].…”

Section: Discussionmentioning

confidence: 99%

Comparison of Very Short Answer Questions and Multiple Choice Questions in Medical Students: Reliability, Discrimination, Acceptability and Effect on Knowledge Retention

Janse

Wijk

Ruijter

et al. 2022

Preprint

View full text Add to dashboard Cite

Introduction: Multiple choice questions (MCQs) offer high reliability and easy machine-marking, but allow for cueing and stimulate recognition-based learning. Very short answer questions (VSAQs) may circumvent these limitations. We investigated VSAQ reliability, discriminative capability, acceptability, and knowledge retention compared to MCQs. Methods: Dutch undergraduate medical students (n=375) were randomised to a formative exam with VSAQs first and MCQs second or vice versa in two courses, to determine reliability and discrimination. Next, acceptability (i.e., VSAQ review time) was determined in the summative exam. Knowledge retention at 2 and 5 months was determined by comparing score increase on the three-monthly progress test (PT) between students tested with VSAQs and students from previous years tested without VSAQs. Results: Reliability (Cronbachs alpha) was 0.74 for VSAQs and 0.57 for MCQs in one course. In the other course, Cronbachs alpha; was 0.87 for VSAQs and 0.83 for MCQs . Discrimination (Rir) was 0.27 vs. 0.17 and 0.43 vs. 0.39 for VSAQs vs. MCQs, respectively. Reviewing time of one VSAQ for the entire student cohort was ± 2 minutes on average. No clear effect on knowledge retention after 2 and 5 months was observed. Discussion: We found increased reliability and discrimination of VSAQs compared to MCQs. Reviewing time of VSAQs was acceptable. The association with knowledge retention was unclear in our study. This study supports and extends positive results of previous studies on VSAQs regarding reliability, discriminative capability, and acceptability in Dutch undergraduate medical students.

show abstract

“…Questions may also be raised on whether the virtual setting allows for demonstration of a candidate's ability as well as it might in person. Although there have yet to be studies into the validity, reliability, acceptability, educational impact and cost of the virtual CASC (all cited as key features of assessing the utility of an examination), 7 or the impact on the trainee experience, it may be helpful to consider the different aspects of the examination that may contribute to its fairness and utility.…”

mentioning

confidence: 99%

The virtual Clinical Assessment of Skills and Competence: the impact and challenges of a digitised final examination

Chu

Sathanandan

2021

BJPsych Bull

View full text Add to dashboard Cite

The COVID-19 pandemic has affected how clinical examinations are conducted, resulting in the Royal College of Psychiatrists delivering the Clinical Assessment of Skills and Competence virtually. Although this pragmatic step has allowed for progression of training, it has come at the cost of a significantly altered examination experience. This article aims to explore the fairness of such an examination, the difference in trainee experience, and the use of telemedicine to consider what might be lost as well as gained at a time when medical education and delivery of healthcare are moving toward the digitised frontier.

show abstract

Transforming MRCPsych theory examinations: digitisation and very short answer questions (VSAQs)

Cited by 5 publications

References 20 publications

Functional near-infrared spectroscopy of medical students answering various item types

Functional near-infrared spectroscopy of medical students answering various item types

Comparison of Very Short Answer Questions and Multiple Choice Questions in Medical Students: Reliability, Discrimination, Acceptability and Effect on Knowledge Retention

The virtual Clinical Assessment of Skills and Competence: the impact and challenges of a digitised final examination

Contact Info

Product

Resources

About