Efficient incremental loading in ETL processing for real-time data integration

Biswas, Neepa; Sarkar, Anamitra; Mondal, Kartick Chandra

doi:10.1007/s11334-019-00344-4

Cited by 35 publications

(18 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since the ETL pipelines for this use case may require the intervention of cohort owners, a tool with collaborative features may simplify their implementation. Kettle, also known as Pentaho Data Integration (PDI), is considered one of the most relevant and complete tools aiming to simplify the design and creation of ETL processes [14,15]. The problem with this tool is the lack of collaborative features.…”

Section: Collaborative Etl Toolmentioning

confidence: 99%

Harmonising Alzheimer's Disease Cohorts using a Common ETL Tool

Almeida

Oliveira

2022

Preprint

View full text Add to dashboard Cite

Background: Many scientific studies have sought to obtain better understanding of specific medical conditions. Concerning Alzheimer’s Disease, there is a lack of reliable diagnostics and this can be related to the availability of only small-scale ongoing biomarker studies and longitudinal cohorts including these subjects. Aiming to generate more substantial clinical evidence, researchers have started to perform multiple cohort analyses. While this is currently possible by harmonising these cohorts into a common data model, the migration pipelines are usually implemented using programming languages. Therefore, cohort owners may have difficulties in contributing during the validation stage of these pipelines.Results: To reduce the dependency on technical teams’ support when validating the data transformations, we propose the use of an ETL tool with visual features. BIcenter is a collaborative web platform designed to implement ETL tasks through the browser. These pipelines are constructed using drag-and-drop features and intuitive forms to customise the ETL steps. This tool is an open-source project and is accessible at https://bioinformatics-ua.github.io/BIcenter/.Conclusions: Our methodology produces interoperable cohorts for multicentric disease-specific studies. Therefore, we validated this using Alzheimer’s Disease cohorts from several countries, combining at the end 6,669 subjects and 172 medical attributes. The harmonised cohorts now enable multi-cohort querying and analysis, helping in the execution of new studies.

show abstract

Section: Collaborative Etl Toolmentioning

confidence: 99%

Harmonising Alzheimer's Disease Cohorts using a Common ETL Tool

Almeida

Oliveira

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Overall, a full process of extracting transforming, and loading (ETL) data from homogeneous or heterogeneous sources [52,53] was established to access our data. In short, it's an essential component in cleansing, customization, reformatting, integration, and insertion of our prerequired data [54,55,56].…”

Section: Score Prioritization (Genementioning

confidence: 99%

Adjustment of p-Value Expression To Ontology Using Machine Learning For Genetic Prediction, Prioritization, Interaction, And Its Validation In Glomerular Disease

Ettetuani

Chahboune

Moussa

2022

Preprint

View full text Add to dashboard Cite

Background: The selection of gene expression mixed to multiple similarity scores of their gene ontology terms (GO), were adjusted by our algorithm as essential prediction tasks for evaluating the regulatory pathways. Moreover applying machine learning techniques, in order to link the gene products into networks that prioritize candidate genes as classification. Finally detecting some significant genetic interactions by applying our algorithm model as validation of the results. Results: Experimental validation of all associations facilitates the discovery of causative genes known to be related to glomerular diseases (GD). As well as EGR1, IL33, BMP2, SLAMF8. In which their GO annotations related to this gene include kidney vasculature development, regulation of cell activation/ inflammatory/immune effector/adaptive immune/glomerulus/glomerular mesangial cell proliferation development, etc. Other genes as TNXA, FCER1A, NME3, FMOD, BTG2, PTGER4, AXL, CYP1A2, CYTL1, BHLHE40, IFI16, SPON1, ETNPPL, COL14A1, ITGAV, MYOZ2, CAMK2A, SORT1, RANBP1, in which their variants information include complement a set of C3, C4, and C7 measurement, serum IgE/IgA measurement, c-reactive protein measurement, nephrotic syndrome, immune system disease, tuberculosis, glomerular filtration rate, chronic kidney disease, etc. The lasts enable a rapid interpretation of complex gene expression studies. As well, other high-throughput genomics assays illustrate an overview of a computational model for gene prioritization and their genetic interactions. Conclusions: The furtherance of combining and adjusting the genes expression to multiple similarity scores of their gene ontology (GO) and interacting all possible cross-association of the results reported classification of genes between among to the subject category and validated the abstracts.

show abstract

“…1) Requirements/challenges for real-time stream processing for real-time DWH Following requirements and challenges for implementation of real-time stream processing for real-time DWH were identified after exploring various studies [4], [14]- [18], [20]- [31], [34]- [36], [39]- [41], [43]- [50], [52], [59], [63]- [70], [78]:…”

Section: Assessment Of Rq3: Which Approaches/tools Have Been Repormentioning

confidence: 99%

Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review

Mehmood

Anees

2020

IEEE Access

View full text Add to dashboard Cite

Contribution: Recently, real-time data warehousing (DWH) and big data streaming have become ubiquitous due to the fact that a number of business organizations are gearing up to gain competitive advantage. The capability of organizing big data in efficient manner to reach a business decision empowers data warehousing in terms of real-time stream processing. A systematic literature review for real-time stream processing systems is presented in this paper which rigorously look at the recent developments and challenges of real-time stream processing systems and can serve as a guide for the implementation of real-time stream processing framework for all shapes of data streams. Background: Published surveys and reviews either cover papers focusing on stream analysis in applications other than real-time DWH or focusing on extraction, transformation, loading (ETL) challenges for traditional DWH. This systematic review attempts to answer four specific research questions. Research Questions: 1)Which are the relevant publication channels for real-time stream processing research? 2) Which challenges have been faced during implementation of real-time stream processing? 3) Which approaches/tools have been reported to address challenges introduced at ETL stage while processing real-time stream for real-time DWH? 4) What evidence have been reported while addressing different challenges for processing real-time stream? Methodology: A systematic literature was conducted to compile studies related to publication channels targeting real-time stream processing/joins challenges and developments. Following a formal protocol, semi-automatic and manual searches were performed for work from 2011 to 2020 excluding research in traditional data warehousing. Of 679,547 papers selected for data extraction, 74 were retained after quality assessment. Findings: This systematic literature highlights implementation challenges along with developed approaches for real-time DWH and big data stream processing systems and provides their comparisons. This study found that there exists various algorithms for implementing real-time join processing at ETL stage for structured data whereas less work for un-structured data is found in this subject matter. INDEX TERMS Real-time stream processing, big data streaming, structured/un-structured data, ETL, systematic literature review Publication source Channel References No. %age IEEE Transactions on Knowledge and Data Engineering

show abstract

Efficient incremental loading in ETL processing for real-time data integration

Cited by 35 publications

References 17 publications

Harmonising Alzheimer's Disease Cohorts using a Common ETL Tool

Harmonising Alzheimer's Disease Cohorts using a Common ETL Tool

Adjustment of p-Value Expression To Ontology Using Machine Learning For Genetic Prediction, Prioritization, Interaction, And Its Validation In Glomerular Disease

Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review

Contact Info

Product

Resources

About