BackgroundThe Emergency Severity Index (ESI) has earned reliability and validity in adult populations but has not been adequately evaluated in pediatric patients. The aim of this study was to assess the reliability of the ESI version 4 and inter-rater reliability measures to evaluate the performance of nurses in the emergency ward.MethodsRaters were part of the same team of pediatric emergency medicine team, including pediatric emergency medicine (PEM) physicians and pediatric triage (PT) nurses. Reliability and agreement rates were measured using kappa statistics. The measurements were compared with the admission rates, readmissions to the PEM division, location of admission and death as outcomes.ResultsInitially, PT nurses rated 20 case scenarios. Further in a prospective cohort study, 1104 children were assigned ESI scores by both nurses and physicians. The ratings of case scenarios showed a kappa value of 0.84. In actual patients, ratings showed high concordance with the physicians’ ratings with the kappa value of 0.82 being in a good agreement with the nurses’ ratings. The main area of discordance was detected in level 4 where 48 cases were triaged in higher levels and 25 were triaged in lower levels. The analysis showed the likelihood of admission clearly increased as the ESI score decreased (p<0.0001). There was a significant correlation between the admission status and triage level in both PT nurses’ and PEM physicians’ ratings (Spearman coefficient=0.374, 0.407; p<0.0001).ConclusionESI scores assigned to the pediatric patients are reliable in the hands of experienced PT nurses and PEM physicians. The very good agreement between PT nurses and PEM physicians, demonstrated in this study, is essential in cooperative work in crowded referral emergency departments and helpful in challenging triage cases.