A Bi-Directional Engine for Transformation of Big Data Between NoSQL and SQL Data

Abstract

Big Data emerges as a solution with the continual and massive growth of different types of data streams at high speed. The customized storage and retrieval of data has indeed become a challenge. Over the last few decades, users of all levels had been more conversant with the relational model and SQL (Structured Query Language). However, several limitations of the relational model has been observed and discussed in the existing literature. As of day, a significant number of the applications are cloud-based or web-based. These handle either structured data (SD), semi-structured data (SSD), or completely unstructured data (USD). The volume of the transactions might even be at the petabytes level. NoSQL have drawn considerable attention for successfully handling massive volumes of heterogeneous data efficiently and irrespective of the type of data. It often requires the conversion of unstructured data into a structured format for possible data analysis, etc. This aims to transform the unstructured data stored in JSON format in the MongoDB data model to SQL structured table data format and vice-versa. The primary rigor of this research work is towards designing an interface facility to change the unstructured data into a structured format for its proper re-usability and simplicity. The focus is to design and develop a bi-directional transformation engine that can simplify the complex data in JSON structure to convert the simplified table structure keeping the normalization features and also to transform the data from normalized SQL format to single collection data of JSON format. Therefore, the proposed bi-directional transformation engine will be an interface between the SQL and NoSQL data model. The algorithms are evaluated using appropriate data. The broader vision of this work is to remove the structural differences between unstructured and structured data in respect of its reuse.

Structure of data in cell biology research

“…It is also necessary to prevent a situation where we would have to change the set data types in the entire data model in all columns that follow each other. Choosing the right data type and its size is the basis of a well-designed database (Hanine et al, 2015;Samanta & Chaki, 2023). Currently, it is necessary to use mathematical-statistical models for the correct interpretation of biological, ecological and medical research.…”