A multi-method analysis of free-text comments from the UK General Medical Council Colleague Questionnaires

Richards, Suzanne H; Campbell, John; Walshaw, Emily; Dickens, Andy; Greco, Michael

doi:10.1111/j.1365-2923.2009.03416.x

Cited by 33 publications

(42 citation statements)

References 11 publications

(27 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This finding aligns with a UK study of free-text comments from faculty on colleagues' overall performances, which reported that higher mean scores correlated with a greater number of positive comments. 24 Research on assessments of medical students 22 and trainee doctors 25 showed similar patterns.…”

Section: Explanations For and Interpretations Of Findingsmentioning

confidence: 66%

Frequency and Determinants of Residents’ Narrative Feedback on the Teaching Performance of Faculty

et al. 2013

View full text Add to dashboard Cite

show abstract

Section: Explanations For and Interpretations Of Findingsmentioning

confidence: 66%

Frequency and Determinants of Residents’ Narrative Feedback on the Teaching Performance of Faculty

et al. 2013

View full text Add to dashboard Cite

show abstract

“…Such analyses of comments within quantitative surveys has been previously used to research health service issues. 32 The first stage of the analysis sought to discover the different ways that participants chose to answer the freetext question and develop a representative thematic framework. After careful reading and data immersion, 33 one researcher (RW) conducted a conventional qualitative content analysis, 34 coding all comments and identifying broad themes within an initial framework.…”

Section: Discussionmentioning

confidence: 99%

Qualitative analysis of patients’ feedback from a PROMs survey of cancer patients in England

et al. 2013

View full text Add to dashboard Cite

ObjectiveThis study examined how free-text comments from cancer survivors could complement formal patient-reported outcome measures (PROMs), as part of the England PROMs survey programme for cancer.DesignA qualitative content analysis was conducted of responses to a single open-ended free-text question placed at the end of the cross-sectional population-based postal questionnaire.SettingIndividuals were identified through three UK Cancer Registries and questionnaires were posted to their home addresses.ParticipantsA random sample of individuals (n=4992) diagnosed with breast, colorectal, non-Hodgkins lymphoma or prostate cancer at 1, 2, 3 and 5 years earlier.Results3300 participants completed the survey (68% response rate). Of these 1056 (32%) completed the free-text comments box, indicating a high level of commitment to provide written feedback on patient experience. Almost a fifth (19%) related experiences of excellent care during the treatment phase, with only 8% reporting negative experiences. This contrasted with experiences of care after primary cancer treatment where the majority were negative. Factors impacting negatively upon patient-reported outcomes included the emotional impact of cancer; poor experiences of treatment and care; comorbidities, treatment side effects, social difficulties and inadequate preparation for a wide range of sometimes long-lasting on-going physical and psychological problems. Mediating factors assisting recovery incorporated both professional-led factors, such as quality of preparation for anticipated problems and aftercare services, and participant-led factors, such as learning from other cancer survivors and self-learning through trial and error. The support of friends and family was also a factor in participants' outcomes.ConclusionsThis analysis of free-text comments complements quantitative analysis of PROMs measure's by illuminating relationships between factors that impact on quality of life (QoL) and indicate why cancer patients may experience significantly worse QoL than the general population. The data suggest more systematic preparation and aftercare for individuals to self-manage post-treatment problems might improve QoL outcomes among cancer survivors.

show abstract

“…Qualitative analysts inductively coded the open-text feedback from the GMC-CQ into 5 themes relating to (1) innovation and openness to change (59/1636 comments, 3.6%); (2) interpersonal skills and caring (432/1636 comments, 26.4%); (3) popularity (131/1636 comments, 8%); (4) professionalism (701/1636 comments, 42.8%); and (5) respect or esteem in which the doctor was held (346/1636 comments, 21.1%) [12]. We refer to these categories throughout the rest of the paper as innovator, interpersonal skills, popularity, professionalism, and respect.…”

Section: Methodsmentioning

confidence: 99%

“…Although doctors’ performance might be best assessed by fellow professionals who know them very well, positive reporting bias in open-text reports may occlude differences in performance [11,12]. The challenge therefore is to classify differences in text that is often positively worded and to use these classifications to signal differences in doctors’ performance.…”

Section: Introductionmentioning

confidence: 99%

Supervised Machine Learning Algorithms Can Classify Open-Text Feedback of Doctor Performance With Human-Level Accuracy

et al. 2017

View full text Add to dashboard Cite

BackgroundMachine learning techniques may be an effective and efficient way to classify open-text reports on doctor’s activity for the purposes of quality assurance, safety, and continuing professional development.ObjectiveThe objective of the study was to evaluate the accuracy of machine learning algorithms trained to classify open-text reports of doctor performance and to assess the potential for classifications to identify significant differences in doctors’ professional performance in the United Kingdom.MethodsWe used 1636 open-text comments (34,283 words) relating to the performance of 548 doctors collected from a survey of clinicians’ colleagues using the General Medical Council Colleague Questionnaire (GMC-CQ). We coded 77.75% (1272/1636) of the comments into 5 global themes (innovation, interpersonal skills, popularity, professionalism, and respect) using a qualitative framework. We trained 8 machine learning algorithms to classify comments and assessed their performance using several training samples. We evaluated doctor performance using the GMC-CQ and compared scores between doctors with different classifications using t tests.ResultsIndividual algorithm performance was high (range F score=.68 to .83). Interrater agreement between the algorithms and the human coder was highest for codes relating to “popular” (recall=.97), “innovator” (recall=.98), and “respected” (recall=.87) codes and was lower for the “interpersonal” (recall=.80) and “professional” (recall=.82) codes. A 10-fold cross-validation demonstrated similar performance in each analysis. When combined together into an ensemble of multiple algorithms, mean human-computer interrater agreement was .88. Comments that were classified as “respected,” “professional,” and “interpersonal” related to higher doctor scores on the GMC-CQ compared with comments that were not classified (P<.05). Scores did not vary between doctors who were rated as popular or innovative and those who were not rated at all (P>.05).ConclusionsMachine learning algorithms can classify open-text feedback of doctor performance into multiple themes derived by human raters with high performance. Colleague open-text comments that signal respect, professionalism, and being interpersonal may be key indicators of doctor’s performance.

show abstract

A multi-method analysis of free-text comments from the UK General Medical Council Colleague Questionnaires

Cited by 33 publications

References 11 publications

Frequency and Determinants of Residents’ Narrative Feedback on the Teaching Performance of Faculty

Frequency and Determinants of Residents’ Narrative Feedback on the Teaching Performance of Faculty

Qualitative analysis of patients’ feedback from a PROMs survey of cancer patients in England

Supervised Machine Learning Algorithms Can Classify Open-Text Feedback of Doctor Performance With Human-Level Accuracy

Contact Info

Product

Resources

About