Accuracy of rating scale interval values used in multiple mini-interviews: a mixed methods study

Bégin, Philippe; Gagnon, Robert; Leduc, Jean-Michel; Paradis, Béatrice; Renaud, Jean‐Sébastien; Beauchamp, Jacinthe; Rioux, Richard; Carrier, Marie-Pier; Hudon, Claire; Vautour, Marc; Ouellet, Annie; Bourget, Martine; Bourdy, Christian

doi:10.1007/s10459-020-09970-1

Cited by 5 publications

(3 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although station content was different between the 2 years, the station formats remained the same. Grading in each station was made using a single 6-point Likert-scale (A to F) referring to a station-specific scoring grid with general anchors (A-Excellent, B-Very Good, C-Good, D-Borderline, E-Obvious gaps, F-Insufficient) and then converted to a numerical value between 0 and 100 using a previouslyvalidated asymmetric scale (A = 100, B = 86.7, C = 69.5, D = 51.2, E = 29.3, F = 0) [28]. Stations 2 & 3 (collaborative stations) had the same scoring grid.…”

Section: Methodsmentioning

confidence: 99%

Are different station formats assessing different dimensions in multiple mini-interviews? Findings from the Canadian integrated French multiple mini-interviews

et al. 2022

Self Cite

View full text Add to dashboard Cite

Background Multiple mini-interviews (MMI) are used to assess non-academic attributes for selection in medicine and other healthcare professions. It remains unclear if different MMI station formats (discussions, role-plays, collaboration) assess different dimensions. Methods Based on station formats of the 2018 and 2019 Integrated French MMI (IFMMI), which comprised five discussions, three role-plays and two collaboration stations, the authors performed confirmatory factor analysis (CFA) using the lavaan 0.6-5 R package and compared a one-factor solution to a three-factor solution for scores of the 2018 (n = 1438) and 2019 (n = 1440) cohorts of the IFMMI across three medical schools in Quebec, Canada. Results The three-factor solution was retained, with discussions, role-plays and collaboration stations all loading adequately with their scores. Furthermore, all three factors had moderate-to-high covariance (range 0.44 to 0.64). The model fit was also excellent with a Comparative fit index (CFI) of 0.983 (good if > 0.9), a Tucker Lewis index of 0.976 (good if > 0.95), a Standardized Root Mean Square Residual of 0.021 (good if < .08) and a Root Mean Square Error of 0.023 (good if < 0.08) for 2018 and similar results for 2019. In comparison, the single factor solution presented a lower fit (CFI = 0.819, TLI = 0.767, SRMR = 0.049 and RMSEA = 0.070). Conclusions The IFMMI assessed three dimensions that were related to stations formats, a finding that was consistent across two cohorts. This suggests that different station formats may be assessing different skills, and has implications for the choice of appropriate reliability metrics and the interpretation of scores. Further studies should try to characterize the underlying constructs associated with each station format and look for differential predictive validity according to these formats.

show abstract

Section: Methodsmentioning

confidence: 99%

Are different station formats assessing different dimensions in multiple mini-interviews? Findings from the Canadian integrated French multiple mini-interviews

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…Rater biases are considered as an error which can potentially be corrected through training. [19] [26] [27] We took an interpretivist approach to qualitatively explore explanations for interviewer behaviours within high stakes selection. Given the lack of theoretically informed literature on professional judgement in interviews, our initial framing prior to collecting the data was based on the work-based assessment literature, [3] [4] [28] [29] [30] [31] and the possibility of multiple "true" performance scores.…”

Section: Interviewer Judgement In the Multiple Mini-interviewmentioning

confidence: 99%

Interviewer judgment-making as a social practice: Insights from an institutional observational study in postgraduate selection

Roberts

Annette

Mossman

et al. 2023

Preprint

View full text Add to dashboard Cite

Background Interviewers’ judgements play a critical role in competency-based assessments for selection such as the multiple-mini-interview. ((MMI) Much of the published research focuses on the psychometrics of selection and the impact of rater subjectivity. Within the context of selecting for entry into specialty postgraduate training, we used an interpretivist and socio-constructivist approach to explore how and why interviewers make judgments in high stakes selection settings whilst participating in a multiple-mini-interview (MMI).Methods We explored MMI interviewers’ work processes through an institutional observational approach, based on the notion that interviewers’ judgements are socially constructed and mediated by multiple factors. We gathered data through document analysis, and observations of interviewer training, candidate interactions with interviewers, and interviewer meetings. Interviews included informal encounters in a large selection centre. Data analysis balanced description and explicit interpretation of the meanings and functions of the interviewers’ actions and behaviours.Results Three themes were developed from the data showing how interviewers make professional judgements, specifically by; ‘Balancing interplay of rules and agency,’ ‘Participating in moderation and shared meaning making; and ‘A culture of reflexivity and professional growth.’ Interviewers balanced the following of institutional rules with making judgment choices based on personal expertise and knowledge. They engaged in dialogue, moderation, and shared meaning with fellow interviewers which enabled their consideration of multiple perspectives of the candidate’s performance. Interviewers engaged in self-evaluation and reflection throughout, with professional learning and growth as primary care physicians and supervisors being an emergent outcome.Conclusion This institutional observational study of interviewers at work in an MMI gives rich insight into the social and contextual nature of judgment-based decision-making in the postgraduate selection process, and emergent outcomes relating to the professional growth of interviewers. Reframing interviewer judgement-making as a social practice provides ways to re-design approaches to interviewer training and selection tool design.

show abstract

“…Bias could come from various sources. They included interview content, the process itself, rater subjectivity, deceptive lies, recall errors, reasonableness bias, intentionality bias, or single-motive bias (Bégin et al, 2021;Small & Cook, 2021).…”

Section: Interviews Less Adopted In China But Highly In International...mentioning

confidence: 99%

Comparison Reviews on E-commerce Research Methods: Suggestions for Post-COVID-19

Zhang

Tian

2022

GES

View full text Add to dashboard Cite

This research aimed to understand the differences in research methods adopted between Chinese and international e-commerce research and provided suggestions and recommendations for further research agenda in e-commerce post-COVID-19. This research studied 4,425 Chinese publications between 2015 and 2019 compared to international reviews in e-commerce research. Twelve findings were discussed: 1) Most of China's e-commerce publications (80.24%) did not state any research method. 2) Mixed methods have been well-adopted in China's e-commence research (35.05%), including quantitative-mixed, qualitative-mixed, and quantitative-qualitative mixed methods. They were highly adopted compared to other studies. 3) Survey methods (27.97%) were predominantly single research - purely quantitative method in Chinese research, but they were still behind international research. 4) Case studies (14.05%) were adopted as the second single method in Chinese research. This finding was significantly different from international views. 5) There were different views on whether laboratory experiments should be considered a research method. 6) Document and text analysis methods were reported by Chinese scholars (6.48%), but less and even not be reported internationally. 7) The focus group method was less adopted internationally. 8) The content analysis method (0.84%) was less adopted in Chinese research and was not reported by international scholars. 9) The observation method (0.6%) was less adopted in Chinese research but more adopted in Ain et al. (2019) (2%). Other international studies reported no studies. 10) Interviews (0.36%) were least adopted in Chinese research but highly reported by international scholars. 11) Field study, secondary, and Delphi methods were only adopted in international research, and 12) The correlational research method was not reported in e-commerce research. The main contributions of this study could be stated: Being valuable for researchers interested in Chinese e-commerce research; Adding research value for comparing reviews in e-commerce research methods.

show abstract

Accuracy of rating scale interval values used in multiple mini-interviews: a mixed methods study

Cited by 5 publications

References 21 publications

Are different station formats assessing different dimensions in multiple mini-interviews? Findings from the Canadian integrated French multiple mini-interviews

Are different station formats assessing different dimensions in multiple mini-interviews? Findings from the Canadian integrated French multiple mini-interviews

Interviewer judgment-making as a social practice: Insights from an institutional observational study in postgraduate selection

Comparison Reviews on E-commerce Research Methods: Suggestions for Post-COVID-19

Contact Info

Product

Resources

About