Assessing work ability – a cross-sectional study of interrater agreement between disability claimants, treating physicians, and medical experts

Dell-Kuster, Salome; Lauper, Simon; Koehler, Joerg; Zwimpfer, Johanna; Altermatt, Benedikt; Zwimpfer, Tibor Andrea; Zwimpfer, Leon; Young, Jim; Bucher, Heiner C.; Nordmann, Alain

doi:10.5271/sjweh.3440

Cited by 11 publications

(12 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Of 4562 potentially relevant citations identified, 101 reports proved potentially eligible after we had screened titles and abstracts. On full text screening, 23 studies, 9 11 22 23 24 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 including four non-English studies, 9 39 40 41 proved eligible for analysis (fig 1 ). All studies were published from 1992 onwards and enrolled disability claimants from 12 countries in Europe, North America, Australia, the Middle East, and northeast Asia.…”

Section: Resultsmentioning

confidence: 99%

“… 9 11 22 23 24 32 34 37 39 43 44 45 46 Three studies including 3729 patients (with 3562 patients from a single centre 33 ) and eight raters (information was lacking from one study 33 ) explored agreement between medical experts and claimant’s treating physicians 33 or independent rehabilitation or occupational health teams with a mandate to care. 38 42 The median number of patients per study was 13.5 (range 1-3562), and the median number of raters per study was 12 (2-103, excluding one study that did not report the number of raters 33 ). All but three studies 24 3 42 used a fully crossed design (that is, all raters evaluated all patients), with a median of 11 patients (range 1-180) per rater and a median of 11.5 raters (2-103) per patient.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Inter-rater agreement in evaluation of disability: systematic review of reproducibility studies

Barth

Boer

Busse

et al. 2017

BMJ

View full text Add to dashboard Cite

Objectives To explore agreement among healthcare professionals assessing eligibility for work disability benefits.Design Systematic review and narrative synthesis of reproducibility studies.Data sources Medline, Embase, and PsycINFO searched up to 16 March 2016, without language restrictions, and review of bibliographies of included studies.Eligibility criteria Observational studies investigating reproducibility among healthcare professionals performing disability evaluations using a global rating of working capacity and reporting inter-rater reliability by a statistical measure or descriptively. Studies could be conducted in insurance settings, where decisions on ability to work include normative judgments based on legal considerations, or in research settings, where decisions on ability to work disregard normative considerations.Teams of paired reviewers identified eligible studies, appraised their methodological quality and generalisability, and abstracted results with pretested forms. As heterogeneity of research designs and findings impeded a quantitative analysis, a descriptive synthesis stratified by setting (insurance or research) was performed.Results From 4562 references, 101 full text articles were reviewed. Of these, 16 studies conducted in an insurance setting and seven in a research setting, performed in 12 countries, met the inclusion criteria. Studies in the insurance setting were conducted with medical experts assessing claimants who were actual disability claimants or played by actors, hypothetical cases, or short written scenarios. Conditions were mental (n=6, 38%), musculoskeletal (n=4, 25%), or mixed (n=6, 38%). Applicability of findings from studies conducted in an insurance setting to real life evaluations ranged from generalisable (n=7, 44%) and probably generalisable (n=3, 19%) to probably not generalisable (n=6, 37%). Median inter-rater reliability among experts was 0.45 (range intraclass correlation coefficient 0.86 to κ−0.10). Inter-rater reliability was poor in six studies (37%) and excellent in only two (13%). This contrasts with studies conducted in the research setting, where the median inter-rater reliability was 0.76 (range 0.91-0.53), and 71% (5/7) studies achieved excellent inter-rater reliability. Reliability between assessing professionals was higher when the evaluation was guided by a standardised instrument (23 studies, P=0.006). No such association was detected for subjective or chronic health conditions or the studies’ generalisability to real world evaluation of disability (P=0.46, 0.45, and 0.65, respectively).Conclusions Despite their common use and far reaching consequences for workers claiming disabling injury or illness, research on the reliability of medical evaluations of disability for work is limited and indicates high variation in judgments among assessing professionals. Standardising the evaluation process could improve reliability. Development and testing of instruments and structured approaches to improve reliability in evaluation of disability are urgently ...

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Inter-rater agreement in evaluation of disability: systematic review of reproducibility studies

Barth

Boer

Busse

et al. 2017

BMJ

View full text Add to dashboard Cite

show abstract

“…Work ability impairment has often been described as resulting from symptoms (Dell-Kuster et al 2014;Huijs et al 2012;Stansfeld et al 1995;Szubert and Sobala 2002). However, it has been found that the symptom itself does not predict work ability (Gatchel et al 1994).…”

Section: Toward a Concept Of Work Capacity Impairmentmentioning

confidence: 98%

Different work capacity impairments in patients with different work-anxieties

Muschalla

2015

Int Arch Occup Environ Health

View full text Add to dashboard Cite

show abstract

“…A key criticism of disability evaluations is the failure of medical experts’ to clearly relate how claimants’ impaired health affects their ability to engage in competitive employment [ 14 – 18 ]. Rather, experts refer to their implicit professional expertise [ 11 , 12 , 19 , 20 ]. This gap is prevalent in work capacity evaluations independent of the underlying health condition, and may be a fundamental source for variation between expert judgments and contribute to low reliability of work capacity evaluations.…”

Section: Introductionmentioning

confidence: 99%

Use of a structured functional evaluation process for independent medical evaluations of claimants presenting with disabling mental illness: rationale and design for a multi-center reliability study

et al. 2016

View full text Add to dashboard Cite

BackgroundWork capacity evaluations by independent medical experts are widely used to inform insurers whether injured or ill workers are capable of engaging in competitive employment. In many countries, evaluation processes lack a clearly structured approach, standardized instruments, and an explicit focus on claimants’ functional abilities. Evaluation of subjective complaints, such as mental illness, present additional challenges in the determination of work capacity. We have therefore developed a process for functional evaluation of claimants with mental disorders which complements usual psychiatric evaluation. Here we report the design of a study to measure the reliability of our approach in determining work capacity among patients with mental illness applying for disability benefits.Methods/DesignWe will conduct a multi-center reliability study, in which 20 psychiatrists trained in our functional evaluation process will assess 30 claimants presenting with mental illness for eligibility to receive disability benefits [Reliability of Functional Evaluation in Psychiatry, RELY-study]. The functional evaluation process entails a five-step structured interview and a reporting instrument (Instrument of Functional Assessment in Psychiatry [IFAP]) to document the severity of work-related functional limitations. We will videotape all evaluations which will be viewed by three psychiatrists who will independently rate claimants’ functional limitations. Our primary outcome measure is the evaluation of claimant’s work capacity as a percentage (0 to 100 %), and our secondary outcomes are the 12 mental functions and 13 functional capacities assessed by the IFAP-instrument. Inter-rater reliability of four psychiatric experts will be explored using multilevel models to estimate the intraclass correlation coefficient (ICC). Additional analyses include subgroups according to mental disorder, the typicality of claimants, and claimant perceived fairness of the assessment process.DiscussionWe hypothesize that a structured functional approach will show moderate reliability (ICC ≥ 0.6) of psychiatric evaluation of work capacity. Enrollment of actual claimants with mental disorders referred for evaluation by disability/accident insurers will increase the external validity of our findings. Finding moderate levels of reliability, we will continue with a randomized trial to test the reliability of a structured functional approach versus evaluation-as-usual.

show abstract

Assessing work ability – a cross-sectional study of interrater agreement between disability claimants, treating physicians, and medical experts

Cited by 11 publications

References 20 publications

Inter-rater agreement in evaluation of disability: systematic review of reproducibility studies

Inter-rater agreement in evaluation of disability: systematic review of reproducibility studies

Different work capacity impairments in patients with different work-anxieties

Use of a structured functional evaluation process for independent medical evaluations of claimants presenting with disabling mental illness: rationale and design for a multi-center reliability study

Contact Info

Product

Resources

About