Accurate Medical Vial Identification Through Mixed Reality: A HoloLens 2 Implementation

Abstract

The accurate identification of medicine vials is crucial for emergency medical services, especially for vials that resemble one another but have different labels, volumes, and concentrations. This study introduces a method to detect vials in real-time using mixed reality technology through Microsoft HoloLens 2. The system is also equipped with an SQL server to manage barcode and vial information. We conducted a comparative analysis of the barcode detection capabilities of the HoloLens 2 camera and an external scanner. The HoloLens 2 effectively identified larger barcodes when they were 20–25 cm away in normal lighting conditions. However, it faced difficulties in detecting smaller barcodes that were consistently detected by the external scanner. The frame rate investigation revealed performance fluctuations: an average of 10.54 frames per second (fps) under standard lighting conditions, decreasing to 10.10 fps in low light and further reducing to 10.05 fps when faced with high barcode density. Resolution tests demonstrated that a screen resolution of 1920 × 1080 yielded the best level of accuracy, with a precision rate of 98%. On the other hand, a resolution of 1280 × 720 achieved a good balance between accuracy 93% and speed. The HoloLens 2 demonstrates satisfactory performance under ideal circumstances; however, enhancements in detecting algorithms and camera resolution are required to accommodate diverse surroundings. This approach seeks to help paramedics make quick and accurate decisions during critical situations and tackle common obstacles such as reliance on networks and human mistakes. Our new approach of a hybrid method that integrates an external Bluetooth scanner with the MR device gives optimal results compared to the scanner-only approach.