Chinese herbal medicine image recognition and retrieval have great potential of practical applications. Several previous studies have focused on the recognition with hand-crafted image features, but there are two limitations in them. Firstly, most of these hand-crafted features are low-level image representation, which is easily affected by noise and background. Secondly, the medicine images are very clean without any backgrounds, which makes it difficult to use in practical applications. Therefore, designing high-level image representation for recognition and retrieval in real world medicine images is facing a great challenge. Inspired by the recent progress of deep learning in computer vision, we realize that deep learning methods may provide robust medicine image representation. In this paper, we propose to use the Convolutional Neural Network (CNN) for Chinese herbal medicine image recognition and retrieval. For the recognition problem, we use the softmax loss to optimize the recognition network; then for the retrieval problem, we fine-tune the recognition network by adding a triplet loss to search for the most similar medicine images. To evaluate our method, we construct a public database of herbal medicine images with cluttered backgrounds, which has in total 5523 images with 95 popular Chinese medicine categories. Experimental results show that our method can achieve the average recognition precision of 71% and the average retrieval precision of 53% over all the 95 medicine categories, which are quite promising given the fact that the real world images have multiple pieces of occluded herbal and cluttered backgrounds. Besides, our proposed method achieves the state-of-the-art performance by improving previous studies with a large margin.

With the increasing popularity of herbal medicine, high standards of the high quality control of herbs becomes a necessity, with the herb recognition as one of the great challenges. Due to the complicated processing procedure of the herbs, methods of manual recognition that require chemical materials and expert knowledge, such as fingerprint and experience, have been used. Automatic methods can partially alleviate the problem by deep learning based herb image recognition, but most studies require powerful and expensive computation hardware, which is not friendly to resource-limited settings. In this paper, we introduce a deep learning-enabled mobile application which can run entirely on common low-cost smartphones for efficient and robust herb image recognition with a quite competitive recognition accuracy in resource-limited situations. We hope this application can make contributions to the increasing accessibility of herbal medicine worldwide.

No abstract