We need to talk about reliability: making better use of test-retest studies for study design and interpretation

Matheson, Granville J.

doi:10.7717/peerj.6918

Cited by 173 publications

(158 citation statements)

References 52 publications

Supporting

Mentioning

150

Contrasting

Unclassified

Order By: Relevance

“…As such, an ICC of 1 indicates perfect measurement reliability with all observed variability being due to true (biological) differences and none to measurement variability (error), while an ICC of 0.5 indicates that the variability is comprised of true differences and measurement error in equal measure. Different interpretations of the ICC exist: as proposed by Portney & Watkins [28] and suggested by Matheson [27], we regard an ICC < 0.5 as low, 0.5-0.75 moderate, 0.75-0.9 good, and > 0.9 excellent. Measurement reliability with an ICC > 0.9 is recommended as a lowest acceptable standard for measurements from which diagnostic decisions are made, ICC > 0.7 for research purposes, with 0.95 and 0.8 considered as adequate, respectively [29].…”

Section: Discussionmentioning

confidence: 82%

“…For statistical analysis, R version 3.4.3, was used with the package relfeas (https://github.com/mathesong/relfeas). [ 18 F]FE-PE2I measurement reproducibility was determined with calculation of repeatability (absolute intrasubject variability, AbsVar; and the minimum detectable difference, MDD) and reliability (intraclass correlation coe cient, ICC), as per recommendation of Weir, Baumgartner, and Matheson [25][26][27]. Absolute variability was calculated as: (test-retest)/(mean test and retest)*100.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Reliability of Dopamine Transporter PET Measurements with [18F]FE-PE2I in patients with Parkinson´s Disease

Kerstens

Fazio

Sundgren

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Background. Reliable quantification of dopamine transporter (DAT), a biomarker for Parkinson’s disease (PD), is essential for diagnostic purposes as well as for evaluation of potential disease-modifying treatment. Due to degeneration of dopaminergic neurons and thus lower expected radioligand binding to DAT, higher measurement variability in PD patients might be expected than earlier reproducibility results in healthy controls. Therefore, we aimed to examine the test-retest properties of [18F]FE-PE2I-PET in PD patients. Methods. Nine patients with PD (Hoehn & Yahr stage < 3) were included (men/women: 6/3; mean age 65.2±6.8y). Each patient underwent two [18F]FE-PE2I-PET measurements within 7–28 days. The outcome measure was non-displaceable binding potential generated using wavelet-aided parametric imaging with cerebellum as reference region. We assessed test-retest performance using estimates of reliability and repeatability. Regions for primary analysis were caudate, putamen, ventral striatum, and substantia nigra. Exploratory analysis was performed for functional subdivisions of the striatum. We also compared the more vs. less-affected side.Results. [18F]FE-PE2I showed absolute variability estimates of 5.3–7.6% in striatal regions and 11% in substantia nigra, and ICCs of 0.74—0.97 (median: 0.91). The absolute variability for functional striatal subdivisions was 6.0–9.6% and ICCs of 0.76–0.91 (median: 0.91). The less affected substantia nigra exhibited greater consistency than the more affected side. According to power calculations based on the current sample size, DAT changes of 5–11% in the striatum and 28% in the substantia nigra can be detected with a power of 0.8 (p<0.0125). Conclusion. DAT-PET measurements with [18F]FE-PE2I in PD patients showed good repeatability and reliability. The slightly lower reliability in the substantia nigra in patients may be explained by lower DAT density and smaller anatomical size. Power calculations suggest that [18F]FE-PE2I PET is a suitable marker for longitudinal DAT decline in PD.Trial registration number: EudraCT 2017-003327-29

show abstract

Section: Discussionmentioning

confidence: 82%

Section: Discussionmentioning

confidence: 99%

Reliability of Dopamine Transporter PET Measurements with [18F]FE-PE2I in patients with Parkinson´s Disease

Kerstens

Fazio

Sundgren

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Although graph measures, such as global efficiency and Gcc, show fair-to-good testretest reliability (Welton et al 2015), a recent meta-analysis showed that edges within a functional connectivity matrix -on the basis of which graph measures are calculated -show poor test-retest reliability (Noble et al 2019). This low test-retest reliability influences statistical power and necessitates the inclusion of larger samples to reach the effect size of interest (Matheson 2019;Zuo et al 2019). It is, however, important to note that test-retest reliability is not the same as validity and the meta-analysis showed that one of the main factors that influenced test-retest reliability was artefact correction (Noble et al 2019); a necessary step during preprocessing to remove motion and other non-neural physiological noise from the data and avoid spurious results (Parkes et al 2018).…”

Section: Discussionmentioning

confidence: 99%

Resting-state network topology and planning ability in healthy adults

Vriend

Wagenmakers

Heuvel

et al. 2019

Preprint

View full text Add to dashboard Cite

Functional magnetic resonance imaging (fMRI) studies have been used extensively to investigate the brain areas that are recruited during the Tower of London (ToL) task. Nevertheless, little research has been devoted to study the neural correlates of the ToL task using a network approach. Here we investigated the association between functional connectivity and network topology during restingstate fMRI and ToL task performance, that was performed outside the scanner. Sixty-two (62) healthy subjects (21-74 years) underwent eyes-closed rsfMRI and performed the task on a laptop. We studied global (whole-brain) and within subnetwork resting-state topology as well as functional connectivity between subnetworks, with a focus on the default-mode, fronto-parietal and dorsal and ventral attention networks. Efficiency and clustering coefficient were calculated to measure network integration and segregation, respectively, at both the global and subnetwork level. Our main finding was that higher global efficiency was associated with slower performance (β = .22, Pbca = .04) and this association seemed mainly driven by inter-individual differences in default-mode network connectivity. The reported results were independent from age, sex, education-level and motion.Although this finding is contrary to earlier findings on general cognition, we tentatively hypothesize that the reported association may indicate that individuals with a more integrated brain during the resting-state are less able to further increase network efficiency when transitioning from a rest to task state, leading to slower responses. This study also adds to a growing body of literature supporting a central role for the default-mode network in individual differences in cognitive performance.

show abstract

“…The intra-class correlation coefficient (ICC) for interrater reliability along with the 95% confidence interval (95% CI) was calculated for all four raters, as well as for the two orthopaedic surgeons and two radiologist raters separately as a subgroup analysis. The two-way random effects model for the single rater was used for inter-rater reliability and ICC with a two-way mixed-effects model (single rater) was calculated for test-retest reliability, as recommended in literature [17][18][19] . The ICC values for PTTG and TTTG were compared.…”

Section: Methodsmentioning

confidence: 99%

A Comparison of the Reliability of the Patellar TendonTrochlear Groove (PTTG) Distance and the Tibial TuberosityTrochlear Groove (TTTG) Distance Measured on MRI

Gupta

Batta²,

Kataria

et al. 2020

Malays Orthop J

View full text Add to dashboard Cite

Introduction: An increased tibial tuberosity-trochlear groove (TTTG) distance is used for deciding a treatment plan in patello-femoral instability (PFI). The centre of the patellar tendon and the chondral trochlear groove can be directly visualised on MRI, and measured, giving the patellar tendontrochlear groove (PTTG) distance. A study was designed to compare the inter-rater and the test-retest reliabilities of PTTG and TTTG measurements in MRI of patients without PFI and in a group with PFI. Materials and Methods:This cross-sectional reliability study was done on archival MRI films of 50 patients without patellar instability and 20 patients with patellar instability. TTTG and PTTG distances were independently measured by two orthopaedic surgeons and two radiologists. A hybrid PTTG measurement with bony landmarks on the femoral side and the patellar tendon landmark on the tibial side, was used to estimate the influence of the differences in the femoral and tibial landmarks on the difference in reliabilities. The intra-class correlation coefficient (ICC) was calculated for all four raters, as well as separately for each rater. Results: The PTTG distance had a higher inter-rater reliability (ICC=0.86, 95% CI=0.79-0.92) compared to the TTTG distance (ICC=0.70, 95% CI=0.59-0.80) in patients without PFI. Similar trends were seen in patients with PFI (0.83 vs 0.66). The inter-rater reliability for the hybrid PTTG distance was found to lie in between the TTTG and PTTG. Conclusions: The MRI-based PTTG distance had better inter-rater reliability compared with the MRI-based TTTG distance.

show abstract

We need to talk about reliability: making better use of test-retest studies for study design and interpretation

Cited by 173 publications

References 52 publications

Reliability of Dopamine Transporter PET Measurements with [18F]FE-PE2I in patients with Parkinson´s Disease

Reliability of Dopamine Transporter PET Measurements with [18F]FE-PE2I in patients with Parkinson´s Disease

Resting-state network topology and planning ability in healthy adults

A Comparison of the Reliability of the Patellar TendonTrochlear Groove (PTTG) Distance and the Tibial TuberosityTrochlear Groove (TTTG) Distance Measured on MRI

Contact Info

Product

Resources

About