Medicine Recognition from Colors and Text

Proma, Tanjina Piash; Hossan, Md. Zakir; Amin, M. Ashraful

doi:10.1145/3338472.3338484

Cited by 10 publications

(7 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…However, the effectiveness of Guo et al's technique is limited by certain factors such as the lighting condition, the camera resolution, and the pill and background color contrast. Some pill recognition techniques have been developed to identify a pill based on only a subset of shape, color, and imprint, such as the works in [36][37][38]. The work in [39] identified pills that had only one of four pre-identified colors and classes.…”

Section: Related Workmentioning

confidence: 99%

CNN-Based Pill Image Recognition for Retrieval Systems

et al. 2023

View full text Add to dashboard Cite

Medication should be consumed as prescribed with little to zero margins for errors, otherwise consequences could be fatal. Due to the pervasiveness of camera-equipped mobile devices, patients and practitioners can easily take photos of unidentified pills to avert erroneous prescriptions or consumption. This area of research goes under the umbrella of information retrieval and, more specifically, image retrieval or recognition. Several studies have been conducted in the area of image retrieval in order to propose accurate models, i.e., accurately matching an input image with stored ones. Recently, neural networks have been shown to be effective in identifying digital images. This study aims to provide an enhancement to image retrieval in terms of accuracy and efficiency through image segmentation and classification. This paper suggests three neural network (CNN) architectures: two models that are hybrid networks paired with a classification method (CNN+SVM and CNN+kNN) and one ResNet-50 network. We perform various preprocessing steps by using several detection techniques on the selected dataset. We conduct extensive experiments using a real-life dataset obtained from the National Library of Medicine database. The results demonstrate that our proposed model is capable of deriving an accuracy of 90.8%. We also provide a comparison of the above-mentioned three models with some existing methods, and we notice that our proposed CNN+kNN architecture improved the pill image retrieval accuracy by 10% compared to existing models.

show abstract

Section: Related Workmentioning

confidence: 99%

CNN-Based Pill Image Recognition for Retrieval Systems

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Ling et al [6] tackled the fewshot pill detection problem with a Multi-Stream (MS) deep learning model. In [12], the authors integrated three handcrafted features, namely shape, color, and imprinted text, to identify pills.…”

Section: Pill Classificationmentioning

confidence: 99%

High accurate and explainable multi-pill detection framework with graph neural network-assisted multimodal data fusion

Nguyen,

Pham,

Trung

et al. 2023

PLoS ONE

View full text Add to dashboard Cite

Due to the significant resemblance in visual appearance, pill misuse is prevalent and has become a critical issue, responsible for one-third of all deaths worldwide. Pill identification, thus, is a crucial concern that needs to be investigated thoroughly. Recently, several attempts have been made to exploit deep learning to tackle the pill identification problem. However, most published works consider only single-pill identification and fail to distinguish hard samples with identical appearances. Also, most existing pill image datasets only feature single pill images captured in carefully controlled environments under ideal lighting conditions and clean backgrounds. In this work, we are the first to tackle the multi-pill detection problem in real-world settings, aiming at localizing and identifying pills captured by users during pill intake. Moreover, we also introduce a multi-pill image dataset taken in unconstrained conditions. To handle hard samples, we propose a novel method for constructing heterogeneous a priori graphs incorporating three forms of inter-pill relationships, including co-occurrence likelihood, relative size, and visual semantic correlation. We then offer a framework for integrating a priori with pills’ visual features to enhance detection accuracy. Our experimental results have proved the robustness, reliability, and explainability of the proposed framework. Experimentally, it outperforms all detection benchmarks in terms of all evaluation metrics. Specifically, our proposed framework improves COCO mAP metrics by 9.4% over Faster R-CNN and 12.0% compared to vanilla YOLOv5. Our study opens up new opportunities for protecting patients from medication errors using an AI-based pill identification solution.

show abstract

“…In addition, they offered a two-stage training technique to solve the data scarcity constraint; the first stage is to train with all samples, while the second concentrates only on the hard examples. In [12], the authors integrated three handcrafted features, namely shape, color, and imprinted text, to identify pills. Specifically, the authors first used statistical measurements from the pill's histogram to estimate the number of colors in the pill.…”

Section: Myonal_50mgmentioning

confidence: 99%

“…To improve identification accuracy, the author incorporated data from numerous sources, including RGB, Texture, Contour, and Imprinted Text. Hand-crafted features such as shape, color, and imprinted text were also used in the [12]. The shortcoming of these approaches is that they handle each pill in the picture separately, without taking use of the pill's interaction.…”

Section: Related Workmentioning

confidence: 99%

Image-based Contextual Pill Recognition with Medical Knowledge Graph Assistance

Nguyen¹,

Nguyen²,

Pham³

et al. 2022

Preprint

View full text Add to dashboard Cite

In many healthcare applications, identifying pills given their captured images under various conditions and backgrounds has been becoming more and more essential. Several efforts have been devoted to utilizing the deep learning-based approach to tackle the pill recognition problem in the literature. However, due to the high similarity between pills' appearance, misrecognition often occurs, leaving pill recognition a challenge. To this end, in this paper, we introduce a novel approach named PIKA that leverages external knowledge to enhance pill recognition accuracy. Specifically, we address a practical scenario (which we call contextual pill recognition), aiming to identify pills in a picture of a patient's pill intake. Firstly, we propose a novel method for modeling the implicit association between pills in the presence of an external data source, in this case, prescriptions. Secondly, we present a walkbased graph embedding model that transforms from the graph space to vector space and extracts condensed relational features of the pills. Thirdly, a final framework is provided that leverages both image-based visual and graph-based relational features to accomplish the pill identification task. Within this framework, the visual representation of each pill is mapped to the graph embedding space, which is then used to execute attention over the graph representation, resulting in a semantically-rich context vector that aids in the final classification. To our knowledge, this is the first study to use external prescription data to establish associations between medicines and to classify them using this aiding information. The architecture of PIKA is lightweight and has the flexibility to incorporate into any recognition backbones. The experimental results show that by leveraging the external knowledge graph, PIKA can improve the recognition accuracy from 4.8% to 34.1% in terms of F1 -score, compared to baselines.

show abstract

Medicine Recognition from Colors and Text

Cited by 10 publications

References 4 publications

CNN-Based Pill Image Recognition for Retrieval Systems

CNN-Based Pill Image Recognition for Retrieval Systems

High accurate and explainable multi-pill detection framework with graph neural network-assisted multimodal data fusion

Image-based Contextual Pill Recognition with Medical Knowledge Graph Assistance

Contact Info

Product

Resources

About