Open data integration

Miller, Renée J.

doi:10.14778/3229863.3240491

Cited by 67 publications

(33 citation statements)

References 67 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…At the schema-level, these techniques usually use the syntactic similarity of the names of the schema components for the matching task. At the instance-level, values are usually compared using Jaccard similarity of intersecting exact values [23]. This can also be achieved by first matching duplicate instances and finding the correspondences between their schema components [8].…”

Section: Related Workmentioning

confidence: 99%

“…The current focus of the schema matching research community is to implement efficient holistic schema matching that improves performance by reducing the number of actual comparisons to conduct [23]. To handle this challenge, multiple techniques were proposed.…”

Section: Related Workmentioning

confidence: 99%

“…As a result, we can see that we propose an approach based not only on string matching, but also on content metadata matching involving statistics and profiles of data stored in the datasets. Content metadata are matched based on approximate similarities and not just exact value-matches at the instance-level [20,23]. We focus on proposing novel earlypruning techniques that use supervised learning to pre-filter irrelevant dataset pairs and to detect likely-to-be similar pairs.…”

Section: Related Workmentioning

confidence: 99%

“…However, it is a challenge for data wranglers [13,16,35] preparing the datasets The final publication is available at ACM via http://dx.doi.org/10.1145/3388870 for analysis to understand their structure and commonalities for DL governance purposes [4]. They must extract those datasets which have related data to be used together in data analysis tasks [20,23]. This is commonly referred to as schema matching, where the aim is to find connection strengths between similar concepts from different pairs of datasets [8,10,21,27].…”

Section: Introductionmentioning

confidence: 99%

“…This is commonly referred to as schema matching, where the aim is to find connection strengths between similar concepts from different pairs of datasets [8,10,21,27]. The large-scale application of such a task to big data repositories like DLs is referred to as holistic schema matching [5,23,27,29], where the goal is to match multiple datasets together considering them all in the matching task.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Keeping the Data Lake in Form

Alserafi

Abelló

Romero

et al. 2020

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

Data Lakes (DLs) are large repositories of raw datasets from disparate sources. As more datasets are ingested into a DL, there is an increasing need for efficient techniques to profile them and to detect the relationships among their schemata, commonly known as holistic schema matching. Schema matching detects similarity between the information stored in the datasets to support information discovery and retrieval. Currently, this is computationally expensive with the volume of state-ofthe-art DLs. To handle this challenge, we propose a novel early-pruning approach to improve efficiency, where we collect different types of content metadata and schema metadata about the datasets, and then use this metadata in early-pruning steps to pre-filter the schema matching comparisons. This involves computing proximities between datasets based on their metadata, discovering their relationships based on overall proximities and proposing similar dataset pairs for schema matching. We improve the effectiveness of this task by introducing a supervised mining approach for effectively detecting similar datasets which are proposed for further schema matching. We conduct extensive experiments on a real-world DL which proves the success of our approach in effectively detecting similar datasets for schema matching, with recall rates of more than 85% and efficiency improvements above 70%. We empirically show the computational cost saving in space and time by applying our approach in comparison to instance-based schema matching techniques.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Keeping the Data Lake in Form

Alserafi

Abelló

Romero

et al. 2020

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

show abstract

Demeter: An automatic framework for data migration in open data lakes

Kim,

Han,

Son

et al. 2023

Softw Pract Exp

View full text Add to dashboard Cite

An open data lake stores various forms and types of open data, and there is an increasing demand to manage raw data in tables rather than files for efficient data exploration and analysis. In this paper, we investigate the data management of open data lakes and recognize the limitations of table migration and related problems. First, open data lakes have problems of preprocessing complexity, scale limitation, and platform dependency due to the traditional data management method and open data characteristics. Second, existing studies for table migration have problems of lack of scalability, migration incompleteness, and scale limitation. In this work, we present a novel automation framework, called Demeter, which solves three problems inherent in open data lakes by expanding automation. Specifically, it supports automating catalog collection and preprocessing tasks to solve preprocessing complexity and scale limitation. It also supports platform universality for representative data platforms through the automation of catalog analysis and detailed processing logic. Demeter then solves three problems in table migration by adopting Airbyte, an open‐source ELT platform, and by enhancing automation capability with the Airbyte manager. We verify that Demeter resolves all the problems above through extensive experiments and proves its scalability and universality. In addition, significantly outperforms CKAN by Demeter up to 508.5% in automation performance, up to 207.28% in processing time, and up to 917.17% in migration performance. These results indicate that Demeter is an excellent automation framework that increases the utilization of large‐scale open data and supports reliable Internet‐scale migration.

show abstract

A survey of biodiversity informatics: Concepts, practices, and challenges

Gadelha

Siracusa

Dalcin

et al. 2020

WIREs Data Min & Knowl

View full text Add to dashboard Cite

The unprecedented size of the human population, along with its associated economic activities, has an ever‐increasing impact on global environments. Across the world, countries are concerned about the growing resource consumption and the capacity of ecosystems to provide resources. To effectively conserve biodiversity, it is essential to make indicators and knowledge openly available to decision‐makers in ways that they can effectively use them. The development and deployment of tools and techniques to generate these indicators require having access to trustworthy data from biological collections, field surveys and automated sensors, molecular data, and historic academic literature. The transformation of these raw data into synthesized information that is fit for use requires going through many refinement steps. The methodologies and techniques applied to manage and analyze these data constitute an area usually called biodiversity informatics. Biodiversity data follow a life cycle consisting of planning, collection, certification, description, preservation, discovery, integration, and analysis. Researchers, whether producers or consumers of biodiversity data, will likely perform activities related to at least one of these steps. This article explores each stage of the life cycle of biodiversity data, discussing its methodologies, tools, and challenges. This article is categorized under: Algorithmic Development > Biological Data Mining

show abstract

Open data integration

Cited by 67 publications

References 67 publications

Keeping the Data Lake in Form

Keeping the Data Lake in Form

Demeter: An automatic framework for data migration in open data lakes

A survey of biodiversity informatics: Concepts, practices, and challenges

Contact Info

Product

Resources

About