Assessment of interobserver variability in a Dutch multicenter study on acute ischemic stroke.

Abstract

Quantitative assessment of patient data is a pertinent part of controlled clinical studies. When several centers are involved, the degree of agreement between different observers becomes important. Therefore, in addition to developing a multicenter study on acute ischemic stroke, we have estimated the interobserver agreement expressed in terms of K statistics. Twelve patients suffering from neurologic deficits due to acute ischemic stroke were examined by four investigators, and the results were assessed using the Mathew scale. Considerable interobserver variability was found. Agreement on items based on subjective information from the patient was low, and it is also possible that this information changes with time. It is advised that in the development of assessment scales, items with low interobserver agreement should be avoided. 4 in what they believed to be consistent findings. After starting a multicenter study on acute ischemic stroke, we decided to determine in a separate study the number of interobserver differences in recording neurologic signs. This article deals with the results of that study. Subjects and MethodsTwelve patients, aged 62-84 years, who were hospitalized in a nursing home consented to participate in our study. All suffered from neurologic deficits as the result of acute ischemic strokes that had occurred 3-11 months earlier. They were selected randomly from patients available in the wards on one day by an independent and nonparticipating physician. The patients were unknown to the four participating investigators, and all patients were considered to be in a stable state.The investigators were the four senior neurologists from the participating centers involved in a multicenter study on acute ischemic stroke who were directly responsible for patient care in their own units. All investigators were trained in clinical neurology, had at least 15 years' experience, and were especially interested in cerebrovascular diseases. The neurologic deficits were assessed using the single items of an ordinal scale developed by Mathew et al, 5 modified in only minor detail for the assessment of language disturbance.6 These items are listed in Figure 1 and were considered separately in the statistical analysis because there may be different levels of agreement for the separate parts of the neurologic investigation represented by the items. Definitions of the terms for different items were predetermined and specified in a protocol.The sequence of the four observers' scoring of each patient was randomized to balance the interaction between investigator and patient ( Figure 2). All evaluations were done the same morning to avoid dayto-day effects.At the beginning of each session, each neurologist was given a case record form containing the scoring system for assessment of neurologic deficits and a list of the patients. The only information that the investigators had was the name and age of the aphasic patients. Each patient was assessed separately by the investigators. The case record form was filled out im...

Interobserver Agreement for the Bedside Clinical Assessment of Suspected Stroke

“…9 Our subjects presented with the undifferentiated clinical syndrome of "brain attack," unlike early studies in which stable stroke patients were used and observers were expecting to find neurological symptoms or signs. [3][4][5]7,10 Observers had different levels of experience, which reflects clinical reality (they were not highly trained stroke neurologists, 3,4 nor were the signs demonstrated to the less experienced members of the team by a stroke neurologist 5 ).…”

“…Many studies have measured the inter-rater reliability of elements of the neurological examination, finding it to vary from fair to good. [2][3][4][5][6][7][8][9][10][11] Only 2 studies measured items of the history, and these found surprisingly poor inter-rater reliability. 2,3 No studies have assessed the inter-rater reliability of the clinical diagnosis of stroke (or stroke mimic).…”

A critical appraisal of stroke evaluation and rating scales.

“…31 A study of 344 patients found that low scores predicted early demise but not stroke subtype. 32 Using a slightly modified version of the Mathew scale, considerable interobserver variability was reported in a 12-patient study by Gelmers et al 33 Agreement among investigators ranged from "slight" (K=0.159) for hemianopsia to almost perfect in areas including reflexes and loss of consciousness. The Mathew scale was found to have a low internal consistency (a=0.54) compared with other measures, 31 perhaps suggesting the presence of more than one construct underlying the scale.…”

Unassisted Assessment of Stroke Severity Using Telemedicine

“…12,19,21,[23][24][25][26][27][28][29][30][31][32][33][34][35][36][37] The various items of these scales were submitted to 5 stroke neurologists (A.D.S., J.D.K., R.B., R.-J.V.H., and S.D.R. ), 2 neurolinguistic experts (P.M. and P.P.…”