On-demand big data integration

Kathiravelu, Pradeeban; Sharma, Ashish; Galhardas, Héléna; Roy, Peter Van; Veiga, Luís

doi:10.1007/s10619-018-7248-y

Cited by 10 publications

(7 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The ETL process can be used to integrate and load data from different sources completely; this process begins with the creation of an integrated repository of data [14]. Tasks involved in the ETL process are data acquisition from more than one source (extraction), data processing according to the warehouse integrity standards (transformation), and data populating in the warehouse in the form of new records (loading) [38].…”

Section: B Data Transformationmentioning

confidence: 99%

“…Traditional ETL is called an eager process because it starts by loading complete data sources in an integrated repository of data [14]. Data are ineffectively managed through the eager ETL mainly because of (1) the high loading time and bandwidth, and (2) data authorization which could be circumvented if all the data source contents are loaded in an integrated data repository; however, this bypass will increase the risk of data breach by allowing data access to users [14]. A number of approaches has been proposed to solve these issues [9], [40].…”

Section: ) Traditional Etlmentioning

confidence: 99%

“…A number of approaches has been proposed to solve these issues [9], [40]. Lazy ETL involves the eager integration and the loading solely of metadata rather than the actual data entry [14]. Traditional ETL techniques are graphical user interface (GUI) to ease the process of moving data from the source to the target system and hand-coded ETL system, which provides high adaptability.…”

Section: ) Traditional Etlmentioning

confidence: 99%

“…Notably, numerous applications ranging from transport planning to crisis risk management depend on BDI for data access and analysis [14]. Many studies in the literature reviewed geospatial big data.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Review on Integrating Geospatial Big Datasets and Open Research Issues

et al. 2021

View full text Add to dashboard Cite

Big data and geographic information systems (GIS) are two technologies that have increasingly influenced many areas in the last 10 years and will continue to improve and help solve serious global problems, such as consequences of climate change or global pandemics. A wide spectrum of GIS applications interacts with the continuous growth of geospatial big data sources to drive precise and informed decisions. Geospatial big data integration is designed to accomplish the compatibility of distinct geospatial datasets regardless of their spatial coverage. The large number of geospatial big data sources demand effective data integration for storing and handling such datasets, which will be used for geospatial data analysis and visualization. For instance, risk management datasets related to healthcare and the environment are heterogeneous and disparate. Obtaining a unified view of such geospatial big datasets is complicated and challenging, especially if we consider problems related to healthcare pandemics and environmental disasters. Hence, before we can attempt to predict and mitigate processes occurring in these domains, we must realize that geospatial big data integration is crucial in consolidating datasets. We explore and discuss issues involved in integrating geospatial big datasets in this study. We then classify big data integration processes into three categories, namely, data warehousing, data transformation and integration methods. Furthermore, several research challenges focused on geospatial big data, big earth data, data warehousing, data transformation and linked data are presented. Lastly, open research issues and emerging trends that require in-depth investigations in the near future are highlighted in this study.INDEX TERMS Big data integration, geographic information system (GIS), geospatial big data.

show abstract

Section: B Data Transformationmentioning

confidence: 99%

Section: ) Traditional Etlmentioning

confidence: 99%

Section: ) Traditional Etlmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Review on Integrating Geospatial Big Datasets and Open Research Issues

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Moreover, when data are loaded for the first time, the complete load approach is employed, involving the complete cleaning of tables. In addition, when adding modified or new records to a target source, an incremental load is employed [12][13][14][15].…”

Section: Introductionmentioning

confidence: 99%

Relational Data Extraction and Transformation: A Study to Enhance Information Systems Performance

Hasan

Bakar

2022

JICCE

View full text Add to dashboard Cite

The most effective method to improve information system capabilities is to enable instant access to several relational database sources and transform data with a logical structure into multiple target relational databases. There are numerous data transformation tools available; however, they typically contain fixed procedures that cannot be changed by the user, making it impossible to fulfill the near-real-time data transformation requirements. Furthermore, some tools cannot build object references or alter attribute constraints. There are various situations in which tool changes in data type cause conflicts and difficulties with data quality while transforming between the two systems. The R-programming language was extensively used throughout this study, and several different relational database structures were utilized to complete the proposed study. Experiments showed that the developed study can improve the performance of information systems by interacting with and exchanging data with various relational databases. The study addresses data quality issues, particularly the completeness and integrity dimensions of the data transformation processes.

show abstract