Comparison of medical student performance in summative undergraduate paediatric examinations and a clinician-designed minimum accepted competency (MAC) assessment

McCrossan, Patrick; O’Donoghue, Dara; Nicholson, Alf J; McCallion, Naomi

doi:10.1186/s12909-021-02642-7

Cited by 3 publications

(9 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Only two-thirds of students passed the MAC exam, whereas 96% of the same students passed their university paediatric examination (2). Their level of paediatric knowledge re ects the university curriculum, therefore there must be signi cant gaps between the university curriculum and the 'hidden' curriculum as determined by non-academic clinicians.…”

Section: Discussionmentioning

confidence: 99%

“…The methodology for designing the content, standard setting and administering the MAC examination has already been described (2). The MAC examination was sat by four different groups; three undergraduate groups (Year 1 Royal College of Surgeons in Ireland (RCSI)), Year 2 RCSI and Queen's University Belfast (QUB)) and one postgraduate group (basic specialist trainees (BST) i.e.…”

Section: Methodsmentioning

confidence: 99%

“…To that end, we designed a novel approach to assessment in which question content was provided by non-faculty paediatric clinicians with the remit of 'must know basic knowledge for a doctor starting work in paediatrics,' (i.e., SHO). This examination was named, Minimum Accepted Competency (MAC) and the methodology and gross examination results have previously been described (2). The logic behind this approach is that we wanted to capture a nite amount of the most relevant crucial knowledge that students would be assessed in, thereby equipping them with the knowledge to tackle most clinical tasks early in in their career and provide a solid platform on which to build.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Minimum Accepted Competency Examination: Test Item Analysis

McCrossan

Nicholson

McCallion

2021

Preprint

Self Cite

View full text Add to dashboard Cite

BackgroundThe minimum accepted competency (MAC) examination in paediatrics was a set of MCQ’s designed to test undergraduate medical students on the most basic, ‘must know’ knowledge as determined by non-faculty paediatric clinicians. The faculty standard set the examination at a relatively difficult 41.2% with only two-thirds of students attaining this score (even though 96% of the same students passed their university paediatric examination). We will now describe the psychometric properties of the examination in a bid to explain these results and provide specific suggestions for future practiceMethodsThe methodology for designing the content, standard setting and administration of the MAC examination has already been described. The examination was set by three groups of undergraduate medical students and paediatric trainee doctors. Test item analysis was performed using difficulty index, discrimination index and Cronbach’s alpha.ResultsTest item difficulty between each group was positively correlated. Correlation of item difficulty with the standard set for each item showed a statistically significant positive relationship. However, for 10 of the items, the mean score did not even reach two standard deviations below the standard set by the faculty. Medical students outperformed the trainee doctors on three items. 18 of 30 items achieved a discrimination index >0.2. Cronbach’s alpha ranged from 0.22-0.59.ConclusionQuestions on respiratory paediatrics and seizures were answered poorly. Despite these topics being major topics within the curriculum, it is possible that the clinical emphasis in these non-faculty derived questions was juxtaposed with the factual recall often required for university examinations. Interestingly, for the questions on which the students performed better than the doctors, the questions were quite specific, and the answer found directly in the curriculum. This reflects how students learn with limited clinical experience and is a good example of assessment being a key driver of learning.The concept of a MAC examination to test basic medical knowledge is feasible and the study presented is an encouraging first step towards this method of assessment. This method is readily transferable to other medical (and allied health) specialties.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Minimum Accepted Competency Examination: Test Item Analysis

McCrossan

Nicholson

McCallion

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…To that end, we designed a novel approach to assessment in which question content was provided by non-faculty paediatric clinicians with the remit of ‘must know basic knowledge for a doctor starting work in paediatrics,’ (i.e., SHO). This examination was named, Minimum Accepted Competency (MAC) and the methodology for creating the test and gross examination results have previously been described [ 2 ]. The logic behind this approach is that we wanted to capture a finite amount of the most relevant crucial knowledge that students should be assessed in, thereby equipping them with the knowledge to tackle most clinical tasks early in in their career and provide a solid platform on which to build.…”

Section: Introductionmentioning

confidence: 99%

“…The average score achieved by undergraduate students was 46% with only two thirds reaching the pass mark, whereas in the university paediatric examination, 96% of students passed. The average score achieved by postgraduate SHO’s was 64% [ 2 ]. Therefore, we concluded that there must be a gap in the expectation of knowledge between non-faculty paediatricians and the academic paediatric faculty.…”

Section: Introductionmentioning

confidence: 99%

Minimum accepted competency examination: test item analysis

2022

Self Cite

View full text Add to dashboard Cite

Background To ascertain if undergraduate medical students attain adequate knowledge to practice in paediatrics, we designed the minimum accepted competency (MAC) examination. This was a set of MCQ’s designed to test the most basic, ‘must know’ knowledge as determined by non-faculty paediatric clinicians. Only two-thirds of undergraduate students passed this exam, despite 96% of the same cohort passing their official university paediatric examination. We aim to describe the psychometric properties of the MAC examination to explore why there was a difference in student performance between these two assessments which should, in theory, be testing the same subject area. We will also investigate if the MAC examination is a potentially reliable method of assessing undergraduate knowledge. Methods The MAC examination was sat by three groups of undergraduate medical students and paediatric trainee doctors. Test item analysis was performed using facility index, discrimination index and Cronbach’s alpha. Results Test item difficulty on the MAC between each group was positively correlated. Correlation of item difficulty with the standard set for each item showed a statistically significant positive relationship. However, for 10 of the items, the mean score achieved by the candidates did not even reach two standard deviations below the standard set by the faculty. Medical students outperformed the trainee doctors on three items. 18 of 30 items achieved a discrimination index > 0.2. Cronbach’s alpha ranged from 0.22–0.59. Conclusion Despite faculty correctly judging that this would be a difficult paper for the candidates, there were a significant number of items on which students performed particularly badly. It is possible that the clinical emphasis in these non-faculty derived questions was juxtaposed with the factual recall often required for university examinations. The MAC examination highlights the difference in the level of knowledge expected of a junior doctor starting work in paediatrics between faculty and non-faculty clinicians and can identify gaps between the current curriculum and the ‘hidden curriculum’ required for real world clinical practice. The faculty comprises physicians in employment by the University whose role it is to design the paediatric curriculum and deliver teaching to undergraduate students. Non-faculty clinicians are paediatric physicians who work soley as clinicians with no affiliation to an educational institution. The concept of a MAC examination to test basic medical knowledge is feasible and the study presented is an encouraging first step towards this method of assessment.

show abstract

Impact of duration and number of subjects on students’ performance in junior clerkship

Aaraj

Kausar

Farooqui

2023

Preprint

View full text Add to dashboard Cite

Background: Medical colleges need curriculum reforms in order to enhance clinical learning contingent on previous observations. Shifa College of medicine (SCM) introduced Family medicine clerkship for final year in year 2022. To accommodate this Paeds and Gynae Obs clerkships were shifted from final to fourth year. This resulted in two major curricular shifts, first: four examining (previously there were three) subjects in 4th professional MBBS exam and Second: 2 years of clinical exposure for Gynae Obs and Paeds (previously it was three years. Methods: This cross-sectional study was conducted from Jan 2021 to June 2022. Annual professional exam results of students of year 2021 and 2022 were included. We analyzed the effect of span of clinical teaching (2 years vs. three years) and the effect of number of examining subjects in professional exam (3 subjects vs. four subjects) on students’ final performance. Data was analysed by SPSS version 23. Mean scores ± SD for all theory and OSCE (Objectively structured clinical exam) were compared. Results were also compared for number of failures and high scores > 80 % between two study cohorts. Results: Two hundred and twenty students were enrolled in the study. No statistically significant difference was found in scores of Ophthalmology theory /OSCE, Gynae & Obs and ENT Theory and Paeds OSCE. Mean scores ± SD were significantly better in ENT OSCE (74.16 ± 6.38 vs. 64.25 ± 7.03, p-value = 0.000) and in Gynae & Obs OSCE (141.59 ± 15.59 vs. 146.66 ± 11.38, p-value 0.015) after 2 years of teaching/ four subjects in professional exam. Paeds theory results showed decline after 2 years of teaching as compared to three years teaching (65.01 ± 6.61 vs. 71.66 ± 7.88, p-value = 0.000). Conclusion: The curricular shift for introduction of Family medicine in senior clerkship with subsequent, reduction in teaching duration and increased number of subjects in 4th professional examination did not significantly affect students’ performance in junior clerkship.

show abstract

Comparison of medical student performance in summative undergraduate paediatric examinations and a clinician-designed minimum accepted competency (MAC) assessment

Cited by 3 publications

References 16 publications

Minimum Accepted Competency Examination: Test Item Analysis

Minimum Accepted Competency Examination: Test Item Analysis

Minimum accepted competency examination: test item analysis

Impact of duration and number of subjects on students’ performance in junior clerkship

Contact Info

Product

Resources

About