DLI-IT: a deep learning approach to drug label identification through image and text embedding

Abstract

Background: Drug label, or packaging insert play a significant role in all the operations from production through drug distribution channels to the end consumer. Image of the label also called Display Panel or label could be used to identify illegal, illicit, unapproved and potentially dangerous drugs. Due to the time-consuming process and high labor cost of investigation, an artificial intelligence-based deep learning model is necessary for fast and accurate identification of the drugs. Methods: In addition to image-based identification technology, we take advantages of rich text information on the pharmaceutical package insert of drug label images. In this study, we developed the Drug Label Identification through Image and Text embedding model (DLI-IT) to model text-based patterns of historical data for detection of suspicious drugs. In DLI-IT, we first trained a Connectionist Text Proposal Network (CTPN) to crop the raw image into sub-images based on the text. The texts from the cropped sub-images are recognized independently through the Tesseract OCR Engine and combined as one document for each raw image. Finally, we applied universal sentence embedding to transform these documents into vectors and find the most similar reference images to the test image through the cosine similarity. Results:We trained the DLI-IT model on 1749 opioid and 2365 non-opioid drug label images. The model was then tested on 300 external opioid drug label images, the result demonstrated our model achieves up-to 88% of the precision in drug label identification, which outperforms previous image-based or text-based identification method by up-to 35% improvement. Conclusion: To conclude, by combining Image and Text embedding analysis under deep learning framework, our DLI-IT approach achieved a competitive performance in advancing drug label identification.

Information Extraction from Product Labels: A Machine Vision Approach

“…This integration aims to provide a robust solution to text extraction and recognition challenges posed by diverse and complex imagery found on product labels. In fact, the visual online detection method is proposed to ensure the label quality in real time during industrial production, indicating the practical application of OCR in industrial settings [23]. Additionally, there is a growing interest in the application of OCR in various industry sectors.…”

“…Additionally, there is a growing interest in the application of OCR in various industry sectors. For instance, a deep learning approach to drug label identification has been proposed to address challenges in drug control and distribution, ensuring the provision of safe and approved drugs to consumers and healthcare professionals [23,24].…”

Automatic Extraction of Medication Information from Cylindrically Distorted Pill Bottle Labels

“…Several studies focus on information extraction from drug and other product labels [6][7][8][9][10][11]. One of the studies proposed a system that can identify unapproved and potentially dangerous medications based on the label using a deep learning approach [6].…”

“…Several studies focus on information extraction from drug and other product labels [6][7][8][9][10][11]. One of the studies proposed a system that can identify unapproved and potentially dangerous medications based on the label using a deep learning approach [6]. X. Liu et al used a Connectionist Text Proposal Network (CTPN) to extract sub-images based on the text followed by OCR; then, the text was vectorized using universal sentence embedding for finding cosine similarity to reference images.…”

A Novel Two-Stage Induced Deep Learning System for Classifying Similar Drugs with Diverse Packaging

“…Nathir A. et al also studied the impact of illumination sources on image recognition in the production process of ampoules [ 22 ]. Liu et al attempted to classify boxed medications by using image cropping and OCR techniques [ 23 ]. Gromova K. et al combined deep convolutional neural networks (DCNN) with optical character recognition (OCR) and natural language processing (NLP) techniques to extract information from the labels of bottled medication, enabling the retrieval of essential label information from images [ 24 ].…”