A Scalable Data Access Layer to Manage Structured Heterogeneous Biomedical Data

Abstract

This work presents a scalable data access layer, called PyEHR, designed to support the implementation of data management systems for secondary use of structured heterogeneous biomedical and clinical data. PyEHR adopts the openEHR's formalisms to guarantee the decoupling of data descriptions from implementation details and exploits structure indexing to accelerate searches. Data persistence is guaranteed by a driver layer with a common driver interface. Interfaces for two NoSQL Database Management Systems are already implemented: MongoDB and Elasticsearch. We evaluated the scalability of PyEHR experimentally through two types of tests, called "Constant Load" and "Constant Number of Records", with queries of increasing complexity on synthetic datasets of ten million records each, containing very complex openEHR archetype structures, distributed on up to ten computing nodes.

Performance Analysis of Not Only SQL Semi-Stream Join Using MongoDB for Real-Time Data Warehousing

“…Authors concluded that Couchbase had better response times than MySQL, especially for larger datasets. Another study by [23] evaluated the scalability of PyEHR, a data access layer, using two NoSQL Database Management Systems: MongoDB and Elasticsearch, for secondary use of structured heterogeneous biomedical and clinical data. However, the un-structured aspect of data is not the focus of these studies.…”