Interrater Agreement on National Institutes of Health Stroke Scale Between Paramedics and Stroke Physicians: Validation Study for the Digital Training Model in the Paramedic Norwegian Acute Stroke Prehospital Project

Abstract

Background Time spent in the prehospital phase of acute stroke care is multifactorial and has an effect on the possibilities for acute treatment. Communication between paramedics and the in-hospital stroke team directly affects time to treatment. A mutual stroke scale such as the National Institutes of Health Stroke Scale (NIHSS) may improve communication quality. The Paramedic Norwegian Acute Stroke Prehospital Project (ParaNASPP) was a stepped-wedge, randomized trial of stroke screening using NIHSS in the ambulance where the intervention was training paramedics in stroke and the NIHSS, with the use of NIHSS made into a mobile app to guide the examination and facilitate communication with the in-hospital stroke team. Objective The aim of this study was to validate the digital training model from the ParaNASPP clinical trial. Methods In total, 24 paramedics were recruited from Oslo University Hospital in Norway to complete the ParaNASPP training model; 20 exclusive videos with predefined NIHSS scores were recorded; and 4 stroke physicians from Oslo University Hospital were included for reference. Bland-Altman plots with 95% limits of agreement (LoA) were calculated—first comparing paramedics and stroke physicians to the predefined scores and then with each other. The predefined LoA were set to 3 points. To align with clinical practice, NIHSS scores were also dichotomized into 2 categories: from 0-5 (minor stroke) or ≥6 (moderate and major stroke), and agreement was calculated using Cohen κ. Results The videos (n=20) had a median (range) NIHSS score of 7 (0-31). The paramedics’ scores were slightly higher than the predefined scores with a mean difference of –0.38 and the LoA ranging from –4.04 to 3.29. The paramedics scored higher than the stroke physicians with a mean difference of –0.39, with the LoA ranging from –4.58 to 3.80. When the NIHSS scores were dichotomized, Cohen κ was 0.89 between the predefined scores and paramedics, 0.92 between the predefined scores and stroke physicians, and 0.81 between the paramedics and stroke physicians, all indicating very good agreement. Conclusions The paramedics scored higher than both the predefined scores and stroke physicians’ scores, hence the predefined LoA were not met. However, the width of the LoA was smaller than seen when experienced neurologists are compared. When the NIHSS scores were dichotomized, the paramedics achieved very good agreement with both the predefined scores and stroke physicians’ scores. This study demonstrates the possibilities for the transfer of clinical competence in digital simulation training.

Gamification of the National Institutes of Health Stroke Scale (NIHSS) for simulation training—a feasibility study

“…Although we found a major difference between the groups in the average number of simulations per student (14 vs. 2.5), this was not associated with a difference in interrater agreement when tested in the proficiency test. It seems that paramedics trained in NIHSS have a high level of accuracy, but tend to overestimate the NIHSS score slightly, as seen in recent studies comparing prehospital NIHSS to in-hospital scores [ 2 , 7 ]. The student’s high accuracy in both groups might be attributed to the validated NIHSS training program [ 7 ].…”

“…All enrolled students received a validated NIHSS training program for paramedics [ 7 ] consisting of a 45-minute lecture and a practical demonstration of the NIHSS examination by an NIHSS-certified physician, followed by a 45-min simulation session in smaller groups where they practised the examination on each other in pairs (Fig. 2 ).…”

“…They were tested simultaneously by observing and scoring video recordings of six NIHSS assessments, where a paramedic examined a fictitious patient acting different scripted stroke cases with a pre-defined NIHSS score. The video recordings have been used in previous validation studies in the ParaNASPP trials and a slight variance (± 3) was deemed acceptable based on these results [ 7 , 8 ]. In the clinical proficiency test, all students used a special form with pictograms, instead of the standard NIHSS text form.…”

“…The National Institutes of Health Stroke Scale (NIHSS) is the most common stroke scale used in-hospital and recent studies have shown that paramedics can be trained in the full NIHSS, to detect and recognise symptoms of any type of stroke [ 2 , 5 , 6 ]. It also provide paramedics and stroke physicians with a common language along the stroke care chain [ 2 , 7 ].…”

“…As part of the Paramedic Norwegian Acute Stroke Prehospital Project (ParaNASPP), an e-learning program with digital classroom training [ 7 ] and a mobile application that guides the paramedic through the NIHSS assessment have been created [ 8 , 9 ]. However, frequent training is necessary to maintain clinical skills and this is often difficult due to infrequent exposure to stroke patients.…”