Simulations of Hadoop/MapReduce-Based Platform to Support its Usability of Big Data Analytics in Healthcare

Abstract

In many hospital systems, new technologies that influence patient data require extensive technical testing before implementation into production. Therefore, to implement, an existing High Performance Computing (HPC) Linux node clusters via WestGrid were used to represent a simulation of patient data benchmarked and cross-referenced with current metadata profiles in operational hospital systems at the Vancouver Island Health Authority (VIHA), Victoria, Canada. Over the tested cross-platform, the data were generated, indexed and stored over a Hadoop Distributed File System (HDFS) to noSQL database (HBase) that represented three billion patient records. The study objective to establish an interactive Big Data Platform (BDA) was successful implemented in that Hadoop/MapReduce technologies formed the framework of the platform distributed with HBase (key-value NoSQL database storage) and generated desired hospital-specific metadata at very extremely large volumes. In fact, the framework over generated HBase data files took a week or a month for one billion (10TB) and three billion (30TB), respectively. Further performance tests retrieved results from simulated patient records with Apache tools in Hadoop's ecosystem. At optimized iteration, HDFS ingestion with HBase exhibited sustained database integrity over hundreds of iterations; however, to complete the bulk loading via MapReduce to HBase required a month. Inconsistencies of MapReduce limited the capacity to generate/replicate data to HBase efficiently. Hospital system based on patient encounter database was very difficult and data profiles were fully representative of complex patient-to-hospital relationships. Our platform is important to lead discovery of useful big data technologies across multiple hospital systems.