Background: Accurate neonatal pain assessment (NPA) is the key to neonatal pain management, yet it is a challenging task for medical staff. This study aimed to analyze the clinical practicability of the artificial intelligence based NPA (AI-NPA) tool for real-world blood sampling. Method: We performed a prospective study to analyze the consistency of the NPA results given by a self-developed automated NPA system and nurses’ on-site NPAs (OS-NPAs) for 232 newborns during blood sampling in neonatal wards, where the neonatal infant pain scale (NIPS) was used for evaluation. Spearman correlation analysis and the degree of agreement of the pain score and pain grade derived by the NIPS were applied for statistical analysis. Results: Taking the OS-NPA results as the gold standard, the accuracies of the NIPS pain score and pain grade given by the automated NPA system were 88.79% and 95.25%, with kappa values of 0.92 and 0.90 (p < 0.001), respectively. Conclusion: The results of the automated NPA system for real-world neonatal blood sampling are highly consistent with the results of the OS-NPA. Considering the great advantages of automated NPA systems in repeatability, efficiency, and cost, it is worth popularizing the AI technique in NPA for precise and efficient neonatal pain management.

Background: Neonatal pain assessment (NPA) represents a huge global problem of essential importance, as a timely and accurate assessment of neonatal pain is indispensable for implementing pain management. Purpose: To investigate the consistency of pain scores derived through video-based NPA (VB-NPA) and on-site NPA (OS-NPA), providing the scientific foundation and feasibility of adopting VB-NPA results in a real-world scenario as the gold standard for neonatal pain in clinical studies and labels for artificial intelligence (AI)-based NPA (AI-NPA) applications. Setting: A total of 598 neonates were recruited from a pediatric hospital in China. Methods: This observational study recorded 598 neonates who underwent one of 10 painful procedures, including arterial blood sampling, heel blood sampling, fingertip blood sampling, intravenous injection, subcutaneous injection, peripheral intravenous cannulation, nasopharyngeal suctioning, retention enema, adhesive removal, and wound dressing. Two experienced nurses performed OS-NPA and VB-NPA at a 10-day interval through double-blind scoring using the Neonatal Infant Pain Scale to evaluate the pain level of the neonates. Intra-rater and inter-rater reliability were calculated and analyzed, and a paired samples t-test was used to explore the bias and consistency of the assessors’ pain scores derived through OS-NPA and VB-NPA. The impact of different label sources was evaluated using three state-of-the-art AI methods trained with labels given by OS-NPA and VB-NPA, respectively. Results: The intra-rater reliability of the same assessor was 0.976–0.983 across different times, as measured by the intraclass correlation coefficient. The inter-rater reliability was 0.983 for single measures and 0.992 for average measures. No significant differences were observed between the OS-NPA scores and the assessment of an independent VB-NPA assessor. The different label sources only caused a limited accuracy loss of 0.022–0.044 for the three AI methods. Conclusion: VB-NPA in a real-world scenario is an effective way to assess neonatal pain due to its high intra-rater and inter-rater reliability compared to OS-NPA and could be used for the labeling of large-scale NPA video databases for clinical studies and AI training.