Defining and tracking medical student self-monitoring using multiple-choice question item certainty

Abstract

Background Self-monitoring is an important component of clinical practice. It is underpinned by the framework of self-efficacy which is concerned with judgments of how well one believes one can achieve or perform a task. This research aimed to develop criteria for adequate self-monitoring, then to measure patterns of self-monitoring, and to explore how these patterns relate to a student’s year in a medical course and to patterns of knowledge. Methods Analysis of individuals’ levels of correctness in answering assessment items and their certainty in correctness may be used to inform assessments of ability to self-monitor. Two criteria were proposed and applied to define adequate self-monitoring. Firstly, increasing proportions correct with increasing levels of certainty. Secondly, having a proportion correct for high certainty responses that was not lower than cohort levels. Student responses in progress tests comprising multiple-choice questions (MCQs) and associated certainty were analysed. Criteria for the presence of adequate self-monitoring and for adequate knowledge were applied to the results of each of four tests conducted over 2 years, and used to categorise patterns of self-monitoring and knowledge. Results Data from 3 year group cohorts totalling 737 students were analysed. The majority (58%) of students demonstrated adequate knowledge and met both criteria for adequate self-monitoring across all four tests. The most advanced year group cohort had the highest rates of adequate knowledge and the highest rates of meeting both self-monitoring criteria. The patterns of self-monitoring were the same as the patterns of knowledge across the four tests for 454 students, but for the remaining 283 the patterns of self-monitoring and knowledge differed. Conclusion Analysis of responses to item level certainty has informed development of a definition of adequate self-monitoring that may be applied to individual student’s responses from a single test, and to track the adequacy of a student’s self-monitoring over time. Patterns of self-monitoring tend to match patterns of knowledge, but not in all cases, suggesting the self-monitoring measure could provide additional information about student ability.

Improving medical residents’ self-assessment of their diagnostic accuracy: does feedback help?

“…Third, the way participants had to rate their confidence gives us only limited information on their thought processes and behaviours in clinical practice. Future studies could use different descriptors of confidence, similar to Tweed et al ( 2020 ), by asking participants whether they need more knowledge or information to make a decision, would like to consult a colleague, or feel confident to make a decision on their own. These options may also help to teach physicians in training that seeking help is a valid an valuable option, too (although also in this case, being well-calibrated would help to avoid unnecessary help-seeking).…”

Using MCQ response certainty to determine how aspects of self‐monitoring develop through a medical course

“…Previous methods for assessing self-monitoring have included analysing response times [9][10][11]15 ; the rates of flagging or deferring responses to questions [9][10][11]15 ; the rates of changing responses to questions 11,16 ; and self-rated certainty per item. 10,[15][16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31] The notion of certainty in decisions can be used to operationalise the need to 'look it up' 1,9 or 'refer this problem to another individual'. 1 With the concept of certainty being based on this operationalisation, self-monitoring for clinicians in practice and medical students making paper-based clinical decisions can be considered to have three concepts: insightfulness, safety and efficiency.…”

Delirium care knowledge in critical care nurses: A multiple‐choice question‐based quiz

“…[2][3][4] Most instruments have a true/false design for assessing delirium care knowledge, the multiple-choice questions (MCQs) are strictly objective and have been widely used in examinations and knowledge tests internationally. 5,6 Little is known regarding whether a useful examination instrument such as an MCQ quiz is reliable for assessing delirium care knowledge.…”

“…Several instruments have been developed for investigating delirium knowledge among ICU health care professionals, but few instruments for which critical care nurses are the primary target population have been comprehensively validated and assessed 2–4 . Most instruments have a true/false design for assessing delirium care knowledge, the multiple‐choice questions (MCQs) are strictly objective and have been widely used in examinations and knowledge tests internationally 5, 6 . Little is known regarding whether a useful examination instrument such as an MCQ quiz is reliable for assessing delirium care knowledge.…”