Automatic Learning Framework for Pharmaceutical Record Matching

Abstract

Pharmaceutical manufacturers need to analyse a vast number of products in their daily activities. Many times, the same product can be registered several times by different systems using different attributes, and these companies require accurate and quality information regarding their products since these products are drugs. The central hypothesis of this research work is that machine learning can be applied to this domain to efficiently merge different data sources and match the records related to the same product. No human is able to do this in a reasonable way because the number of records to be matched is extremely high. This paper presents a framework for pharmaceutical record matching based on machine learning techniques in a big data environment. The proposed framework aims to explode the well-known rules for the matching of records from different databases for training machine learning models. Then the trained models are evaluated by predicting matches with records that do not follow these known rules. Finally, the production environment is simulated by generating a huge amount of combinations of records and predicting the matches. The obtained results show that, despite the good results obtained with the training datasets, in the production environment, the average accuracy of the best model is around 85%. That shows that matches which do not follow the known rules can be predicted and, considering that there is not a human way to process this amount of data, the results are promising.

Integration and classification approach based on probabilistic semantic association for big data

“…It was therefore not easy to find an effective knowledge model and it was very difficult to manage and group the relationships between the data attributes. Existing integration and analytical frameworks like pharmacy data record matching framework [1], biodiversity data retrieval framework [4] and, data records with schema matching framework [7] are facing many challenges in analyzing this large-scale data due to the complexity of the distribution of data and the proliferation of multi-source data. To solve this problem, and integration and classification model based on the Probability Semantic Association (PSA) of the attribute and generation of knowledge pattern is proposed for a large data source.…”

“…Applications that involve multiple sources, such as Google Base or in-depth web or tools B Dharmendra Singh Rajput dharmendrasingh@vit.ac.in Vishnu VandanaKolisetty kvishnu.vandana2016@vitstudent.ac.in 1 SCOPE, Vellore Institute of Technology, Vellore 632014, India 2 SITE, Vellore Institute of Technology, Vellore 632014, India require the automatic deletion of the semantic match between the intermediation schema and the data sources, which can be approximated. Such as, in [1] a framework for learning the records of pharmacy automatically and in [2] a mechanism to simplify the complex traffic information through autonomous coordinated control in big data.…”

Entity Matching with Similarity Encoding: A Supervised Learning Recommendation Framework for Linking (Big) Data

“…The recent literature on Entity Matching (EM) in finance, economic history, medical research, and computer science makes significant efforts in developing semi-automated frameworks that reduce the human-in-the-loop requirements and enhance access to EMdriven research (Helgertz et al, 2022;P. Li et al, 2021;Abramitzky et al, 2020;López-Cuadrado et al, 2020;Antoni & Schnell, 2019;González-Carrasco et al, 2019;Ebraheem et al, 2018;Mudgal et al, 2018;Rodriguez-Lujan & Huerta, 2016). Nevertheless, these frameworks are accompanied by high technical burdens, and their implementation requires either fitting a comprehensive toolbox of Machine Learning (ML) models and selecting the most accurate (e.g., López-Cuadrado et al, 2020;González-Carrasco et al, 2019) or selecting among vocabularies and using word embeddings to fit an appropriate language model (e.g., López-Cuadrado et al, 2020;Ebraheem et al, 2018).…”