Real-time data warehouse loading methodology and architecture: a healthcare use case

Bouali, Hanen; Akaichi, Jalel; Gaaloul, Ala

doi:10.1504/ijdats.2019.103757

Cited by 4 publications

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similarly, a Data on Demand ETL (DOD-ETL) model is proposed in Machado et al Dec. ( 2019 ), which combines on demand data stream and pipelines in distributed and parallel memory caches to support effective portioning of data. An event-driven architecture is presented in Rieke et al ( 2018 ), to manage spatial data obtained from different capturing devices to maintain the geospatial information according to the real-time data (Bouali et al 2019 ). All the approaches suffer to achieve higher performance in extract transform and loading process in high-dimensional data environments.…”

Section: Introductionmentioning

confidence: 99%

RETRACTED ARTICLE: Dynamic multi-variant relational scheme-based intelligent ETL framework for healthcare management

Manickam

Minu

2022

Soft Comput

View full text Add to dashboard Cite

The growth of information technology has opened the gate for the organizations to maintain their data in various forms and at various volumes. This increases the volume and dimension of data being maintained. However, they store their data in their data servers or in cloud environment. Such data have been used to generate various intelligence to support various problems. To support such analysis process, different data have been used and the big data comes to play in this part. Optimizing techniques to help improve the process of ETL could greatly help in real-time analysis of data. ETL optimization could be achieved through several factors simplest being increasing the frequency of the process. Other ways to achieve optimization are through the use of various architectures, programming models, intelligence in transformation and security. To improve the performance of ETL, an efficient dynamic multi-variant relational intelligent ETL framework has been presented in this article. The distributed approach maintains various ontology’s and data dictionaries which have been dynamically updated by different threads of ETL process. Initially, the process is start by applying the extraction process which extracts the data from different sources and finds set of dimensions and their characteristics. Such data extracted have been verified over the data dictionary. Further, the relational score has been measured for each data source with the existing one. Similarly, the method computes the value of multi-variant relational similarity (MVRS) for the data obtained from a single source. This will be performed by different threads of ETL process. According to the value of MVRS, the method performs map reduce and merging of data. According to the value of MVRS the method selects the data node and merges the data to store in the data warehouse. The threads of ETL are capable of reading the changes in data dictionaries and ontology’s to iterate the process of transformation and loading. The method improves the performance of ETL with least time complexity and higher performance.

show abstract

Section: Introductionmentioning

confidence: 99%

RETRACTED ARTICLE: Dynamic multi-variant relational scheme-based intelligent ETL framework for healthcare management

Manickam

Minu

2022

Soft Comput

View full text Add to dashboard Cite

show abstract

Game theory in network security for digital twins in industry

Feng

Chen

et al. 2024

Digital Communications and Networks

View full text Add to dashboard Cite

Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review

Mehmood

Anees

2020

IEEE Access

View full text Add to dashboard Cite

Contribution: Recently, real-time data warehousing (DWH) and big data streaming have become ubiquitous due to the fact that a number of business organizations are gearing up to gain competitive advantage. The capability of organizing big data in efficient manner to reach a business decision empowers data warehousing in terms of real-time stream processing. A systematic literature review for real-time stream processing systems is presented in this paper which rigorously look at the recent developments and challenges of real-time stream processing systems and can serve as a guide for the implementation of real-time stream processing framework for all shapes of data streams. Background: Published surveys and reviews either cover papers focusing on stream analysis in applications other than real-time DWH or focusing on extraction, transformation, loading (ETL) challenges for traditional DWH. This systematic review attempts to answer four specific research questions. Research Questions: 1)Which are the relevant publication channels for real-time stream processing research? 2) Which challenges have been faced during implementation of real-time stream processing? 3) Which approaches/tools have been reported to address challenges introduced at ETL stage while processing real-time stream for real-time DWH? 4) What evidence have been reported while addressing different challenges for processing real-time stream? Methodology: A systematic literature was conducted to compile studies related to publication channels targeting real-time stream processing/joins challenges and developments. Following a formal protocol, semi-automatic and manual searches were performed for work from 2011 to 2020 excluding research in traditional data warehousing. Of 679,547 papers selected for data extraction, 74 were retained after quality assessment. Findings: This systematic literature highlights implementation challenges along with developed approaches for real-time DWH and big data stream processing systems and provides their comparisons. This study found that there exists various algorithms for implementing real-time join processing at ETL stage for structured data whereas less work for un-structured data is found in this subject matter. INDEX TERMS Real-time stream processing, big data streaming, structured/un-structured data, ETL, systematic literature review Publication source Channel References No. %age IEEE Transactions on Knowledge and Data Engineering

show abstract

Real-time data warehouse loading methodology and architecture: a healthcare use case

Cited by 4 publications

References 0 publications

RETRACTED ARTICLE: Dynamic multi-variant relational scheme-based intelligent ETL framework for healthcare management

RETRACTED ARTICLE: Dynamic multi-variant relational scheme-based intelligent ETL framework for healthcare management

Game theory in network security for digital twins in industry

Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review

Contact Info

Product

Resources

About