Orchid: Integrating Schema Mapping and ETL

Deßloch, Stefan; Hernández, Mauricio A.; Wisnesky, Ryan; Radwan, Ahmed; Zhou, Jindan

doi:10.1109/icde.2008.4497540

Cited by 45 publications

(21 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Applications This is the first algorithm that enables the generation of solutions for mapping scenarios with egds in a scalable way. To see the relevance of this achievement, consider that schema-mappings have been identified as a key component of several classes of applications that exchange or integrate data, for instance ETL [8], object-relational mapping [19], data fusion [5], and schema-integration [22]. It can also be seen that key constraints -and therefore egds -play a very important role in all of these applications.…”

Section: Figure 1: Mapping Person Datamentioning

confidence: 99%

Scalable data exchange with functional dependencies

2010

View full text Add to dashboard Cite

The recent literature has provided a solid theoretical foundation for the use of schema mappings in data-exchange applications. Following this formalization, new algorithms have been developed to generate optimal solutions for mapping scenarios in a highly scalable way, by relying on SQL. However, these algorithms suffer from a serious drawback: they are not able to handle key constraints and functional dependencies on the target, i.e., equality generating dependencies (egds). While egds play a crucial role in the generation of optimal solutions, handling them with first-order languages is a difficult problem. In fact, we start from a negative result: it is not always possible to compute solutions for scenarios with egds using an SQL script. Then, we identify many practical cases in which this is possible, and develop a best-effort algorithm to do this. Experimental results show that our algorithm produces solutions of better quality with respect to those produced by previous algorithms, and scales nicely to large databases.

show abstract

Section: Figure 1: Mapping Person Datamentioning

confidence: 99%

Scalable data exchange with functional dependencies

2010

View full text Add to dashboard Cite

show abstract

“…Such a generic approach thus supports High automation in creating different schema mappings for both data integration and data exchange settings. [19] went further to provide the inter-operability between tools for creating declarative schema mappings (e.g., Clio) and procedural data-intensive tools (e.g., ETL). Still, such schema mappings either cannot tackle grouping and aggregation or overlook complex transformations typical in today's ETL processes.…”

Section: Automationmentioning

confidence: 99%

“…It spans from the works that propose modeling approaches to standardize the design of data flows, especially in the context of ETL processes (Low), e.g., [99,92,3]; then approaches that provide guidelines and/or frequently used patterns to facilitate the design of a data-intensive flow (Medium), e.g., [53,98]; and approaches that attempt to fully automate the generation of dataintensive flows as well as their optimization (High), e.g., [79,24,19].…”

Section: Data Transformationmentioning

confidence: 99%

A Unified View of Data-Intensive Flows in Business Intelligence Systems: A Survey

Jovanovic

Romero

Abelló

2016

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Data-intensive flows are central processes in today's business intelligence (BI) systems, deploying different technologies to deliver data, from a multitude of data sources, in user-preferred and analysisready formats. To meet complex requirements of next generation BI systems, we often need an effective combination of the traditionally batched extract-transform-load (ETL) processes that populate a data warehouse (DW) from integrated data sources, and more real-time and operational data flows that integrate source data at runtime. Both academia and industry thus must have a clear understanding of the foundations of dataintensive flows and the challenges of moving towards next generation BI environments. In this paper we present a survey of today's research on data-intensive flows and the related fundamental fields of database theory. The study is based on a proposed set of dimensions describing the important challenges of data-intensive flows in the next generation BI setting. As a result of this survey, we envision an architecture of a system for managing the lifecycle of data-intensive flows. The results further provide a comprehensive understanding of data-intensive flows, recognizing challenges that still are to be addressed, and how the current solutions can be applied for addressing these challenges.

show abstract

“…An interesting related system is Orchid [23]. It adopts an abstract data model (OHM ) for ETL flows, whose instances are then translated first into schema mappings and then into executable ETL scripts.…”

Section: Scaling In Statistical Data Processingmentioning

confidence: 99%

“…It is also worth mentioning that while most of the theory of mappings has been developed for the relational model, arguments for more general, model-independent approaches have been made [7]. They also have proven applicability: for example, the close relationship between schema mappings and ETL (Extract -Transform -Load) executable flows, typically used in data warehousing, has been practically clarified by many works, e.g., the Orchid prototype [23], and had been understood since Clio early studies [31].…”

mentioning

confidence: 99%

Executable schema mappings for statistical data processing

Atzeni

Bellomarini

Bugiotti

et al. 2017

Distrib Parallel Databases

View full text Add to dashboard Cite

Data processing is the core of any statistical information system. Statisticians are interested in specifying transformations and manipulations of data at a high level, in terms of entities of statistical models. We illustrate here a proposal where a high-level language, EXL, is used for the declarative specification of statistical programs, and a translation into executable form in various target systems is available. The language is based on the theory of schema mappings, in particular those defined by a specific class of tgds, which we actually use to optimize user programs and facilitate the translation towards several target systems. The characteristics of such class guarantee good tractability properties and the applicability in Big Data settings. A concrete implementation, EXLEngine, has been carried out and is currently used at the Bank of Italy.

show abstract

Orchid: Integrating Schema Mapping and ETL

Cited by 45 publications

References 10 publications

Scalable data exchange with functional dependencies

Scalable data exchange with functional dependencies

A Unified View of Data-Intensive Flows in Business Intelligence Systems: A Survey

Executable schema mappings for statistical data processing

Contact Info

Product

Resources

About