Do research studies in the UK reporting child neurodevelopment adjust for the variability of assessors: a systematic review

Abstract

AIM Neurodevelopment is a key outcome for many childhood trials and observational studies. Clinically important decisions may rest on finding relatively small differences in neurodevelopment between groups receiving complex and costly interventions. Our purpose was to determine whether studies which measure neurodevelopment report the numbers, training, and auditing of assessors and, for multiple assessor studies, whether the results were adjusted and if so by which method?METHOD Electronic searches were conducted using Medline, Embase, Cinahl, PsycINFO, and the Cochrane Library. A study was eligible if it reported neurodevelopmental outcome in children resident in the UK, less than or equal to 18 years and was published between 2000 and 2015. Trials and observational studies were included.RESULTS Three hundred and seven full papers were reviewed: 52% of papers did not report the number of assessors used; 21% used a single assessor; and 27% used multiple assessors. Thirty-five per cent mentioned that assessors were trained in the use of the neurodevelopmental tool; 13% of assessors were audited; and only 1% of studies adjusted statistically for the number of assessors.

Inter‐assessor variability: scant data proves the point

“…Study designs frequently maximise participant numbers in order to investigate the possibility of subtle neuropsychological differences, which may be important to long-term outcomes, and because good follow-up rates are important for data validity in longitudinal studies. This has logistical implications, because many studies will require multiple assessors to conduct the assessments, with some potentially problematic consequences, as elucidated in this issue by Khalid et al 1 Standardised assessments are chosen for research partly because the important issues of reliability and validity are included in the assessment design, with assessment manuals incorporating instructions on how to conduct an assessment to meet standardisation requirements. However, there is inevitably some variation in interpersonal exchange between different assessors and study participants.…”

“…Khalid et al 1 found a low rate of reporting of assessor numbers, training and auditing, which may imply a laissezfaire attitude on the part of the authors, reflect a deference to word count, or result from an entrenched blind-spot in reporting. It cannot be assumed that good research practice was not followed.…”

“…2 Despite these initiatives, the concerns raised by Khalid et al 1 have continued to hide in plain sight. Alongside more rigorous reporting guidelines, there has been an increase in statistical reporting and complexity, 3 which raises the question of whether we have not, to some extent, blinded ourselves with these new requirements.…”

“…However, time and budgetary constraints do not relieve us of the responsibility to ensure and report appropriate processes. The advice offered by Khalid et al 1 -that researchers address inter-assessor variability as either meaningful or needing to be adjusted for -evokes Jacob Cohen's summary of his learnings: 'that the informed judgment of the investigator is the crucial element in the interpretation of data, and that things take time'. 4 Perhaps this is a further key requirement: to take time to consider the full range of factors that affect assessment data before any interpretation is made, reported and added to the neurodevelopmental literature.…”

“…Published guidelines designed to improve the reporting of random controlled trials (CONSORT Statement) and observational studies (STROBE Statement) have been widely adopted by researchers and medical journals, with regular updates of additional matters for consideration in an increasing range of specialist areas. 2 Despite these initiatives, the concerns raised by Khalid et al 1 have continued to hide in plain sight. Alongside more rigorous reporting guidelines, there has been an increase in statistical reporting and complexity, 3 which raises the question of whether we have not, to some extent, blinded ourselves with these new requirements.…”