Audio-digital recordings used for independent confirmation of site-based MADRS interview scores

Abstract

Signal detection requires ratings reliability throughout a clinical trial. The confirmation of site-based rater scores by a second, independent and blinded rater is a reasonable metric of ratings reliability. We used audio-digital pens to record site-based interviews of the Montgomery-Asberg Depression Rating Scale (MADRS) in a double-blind, placebo controlled trial of a novel antidepressant in treatment resistant depressed patients. Blinded, site-independent raters generated "dual" scores that revealed high correlations between site-based and site-independent raters (r=0.940 for all ratings) and high sensitivity, specificity, predictive values, and kappa coefficients for treatment response and non-response outcomes using the site-based rater scores as the standard. The blinded raters achieved an 89.4% overall accuracy and 0.786 kappa for matching the treatment response or non-response outcomes of the site-based raters. A limitation of this method is that independent ratings depend on the quality of site-based interviews and patient responses to the site-based interviewers. Nonetheless, this quality assurance strategy may have broad applicability for studies that use subjective measures and wherever ratings reliability is a concern. "Dual" scoring of recorded site-based ratings can be a relatively unobtrusive surveillance strategy to confirm scores and to identify and remediate rater "outliers" during a study.

Audio-digital recordings for surveillance in clinical trials of major depressive disorder

“…Data for this ratings reliability analysis was obtained from 5 phase II or III clinical trials conducted between 2009 and 2017 as part of vendor grants awarded to Clintara LLC (or Bracket LLC) to conduct quality assurance/surveillance programs for ratings precision. The analysis was limited to double-blind, placebo-controlled trials that included patients with major depressive disorder (MDD) that used the MADRS as the primary efficacy measure and had obtained paired (“dual”) site-independent scores based upon audio-digital recordings of site-based interviews [ 16 , 17 ]. All enrolled subjects met DSM-IV criteria for MDD [ 20 , 21 ].…”

“…The use of remote, site-independent (centralized) raters in lieu of site-based raters has been suggested as an alternative method to optimize ratings in a clinical trial [ 4 , [9] , [10] , [11] , [12] , [13] , [14] ]. Alternatively, site-independent review and scoring of site-based ratings has also been introduced as a quality assurance, surveillance strategy to monitor and assure ratings precision by site-based raters during a clinical trial [ [15] , [16] , [17] , [18] ]. This strategy employs audio-digital recording and scoring of site-based interviews to obtain “paired” scores based upon the same interview.…”

ITI-007 for the Treatment of Schizophrenia: A 4-Week Randomized, Double-Blind, Controlled Trial

“…Individuals were required to have shown previous treatment response to APD therapy and thus be neither treatment-naïve nor treatment-resistant. Diagnosis and symptom severity at screening were rated by study investigators and confirmed by independent psychiatrists or clinical psychologists to ensure inclusion of an appropriate patient population (Clintara LLC, Boston, Massachusetts) (17). Patients who met any of the following exclusion criteria were not included in the study: unable to provide informed consent; pregnant/breastfeeding; dementia/delirium/mental retardation/ epilepsy/drug-induced psychosis/brain trauma; schizoaffective disorder/bipolar disorder/acute mania/major depression with psychotic features; imminent danger to self or others; suicidal ideation/behavior; unstable living environment; use of depot antipsychotic within one treatment cycle before baseline; use of any APD within screening period; use of specific agents with known interaction with 5-HT2A receptors; clinically significant abnormal laboratory values or clinical findings; uncontrolled angina/recent history of myocardial infarction/clinically significant cardiac arrhythmia; hematological/renal/hepatic/endocrinological/neurological/cardiovascular disease; history of neuroleptic malignant syndrome; human immunodeficiency virus; hepatitis B or C with evidence of active liver disease; substance abuse/dependence; positive drug/alcohol screen; likely drug allergy/sensitivity; prior participation in a study with ITI-007 or recent exposure to any investigational product; unable to be safely discontinued from current antipsychotic or other psychotropic medications; and any patient judged by the investigator to be inappropriate for study participation.…”

Ratings surveillance and reliability in a study of major depressive disorder with subthreshold hypomania (mixed features)

“…The study used a structured version of the MADRS (SIGMA: the structured interview guide for the MADRS) and the YMRS interview (developed by Sachs, & Harrington, ) in order to foster standardization and improve IRR (Sachs, & Harrington, ; Williams & Kobak, ). Site‐based raters were also trained to use audio‐digital recording devices (“pens”) for the MADRS and YMRS interviews (Targum et al, ; Targum & Pendergrass, ). The pens simultaneously recorded the audio portion of the interview while digitally capturing all written notes on specially manufactured paper.…”

“…Recently, site‐independent raters have been used to assess the reliability of ratings in clinical trials (Leigh‐Pemberton et al, ; Schoemaker, Gaur, Chawla, Jansen, & Szegedi, ; Shen, Kobak, Zhao, Alexander, & Kane, ; Targum et al, ; Targum & Pendergrass, ; Targum, Pendergrass, Toner, Asgharneshad, & Burch, ). One method uses audio‐digital recordings of the site‐based interviews to generate paired site‐independent scores for assessment of rating precision (Targum et al, ; Targum et al, ; Targum & Pendergrass, ). The paired scores can confirm individual site‐based ratings, evaluate rater performance, and identify site‐based ratings that markedly deviate from the site‐independent score (“outliers”).…”