Efficient development of data migration transformations

Carreira, Paulo; Galhardas, Héléna

doi:10.1145/1007568.1007692

Cited by 15 publications

(16 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Practical evidence gathered from the Ajax [8] data cleaning prototype and from Data Fusion [3] legacy-data migration tool, which has been used commercially in large legacy-data migration projects, corroborates the need of supporting data transformations that require one-tomany mappings. Since one-to-many data transformations are not expressible through standard RA queries, we proposed the mapper operator.…”

Section: Discussionmentioning

confidence: 90%

See 1 more Smart Citation

Data Mapper: An Operator for Expressing One-to-Many Data Transformations

Carreira¹,

Galhardas²,

Pereira³

et al. 2005

Data Warehousing and Knowledge Discovery

Self Cite

View full text Add to dashboard Cite

Abstract. Transforming data is a fundamental operation in application scenarios involving data integration, legacy data migration, data cleaning, and extract-transform-load processes. Data transformations are often implemented as relational queries that aim at leveraging the optimization capabilities of most RDBMSs. However, relational query languages like SQL are not expressive enough to specify an important class of data transformations that produce several output tuples for a single input tuple. This class of data transformations is required for solving the data heterogeneities that occur when source data represents an aggregation of target data. In this paper, we propose and formally define the data mapper operator as an extension of the relational algebra to address one-to-many data transformations. We supply an algebraic rewriting technique that enables the optimization of data transformation expressions that combine filters expressed as standard relational operators with mappers. Furthermore, we identify the two main factors that influence the expected optimization gains.

show abstract

Section: Discussionmentioning

confidence: 90%

“…More recently, three efforts, Potter's Wheel [15], Ajax [7] and Data Fusion [3], have proposed operators for data transformation and cleaning purposes.…”

Section: Related Workmentioning

confidence: 99%

Data Mapper: An Operator for Expressing One-to-Many Data Transformations

Carreira¹,

Galhardas²,

Pereira³

et al. 2005

Data Warehousing and Knowledge Discovery

Self Cite

View full text Add to dashboard Cite

show abstract

“…Examples are a book about SAP's data migration tools [14] or explanations of how to migrate to a new product version or how to get away from a competitor's product [15,16]. Furthermore, Carreira and Galhardas describe a specific language designed especially for data migration purposes [17].…”

Section: Related Workmentioning

confidence: 99%

Towards the Industrialization of Data Migration: Concepts and Patterns for Standard Software Implementation Projects

Haller¹

2009

Notes on Numerical Fluid Mechanics and Multidisciplinary Design

View full text Add to dashboard Cite

Abstract. When a bank replaces its core-banking information system, the bank must migrate data like accounts from the old into the new system. Migrating data is necessary but not a catalyst for new business opportunities. The consequence is cost pressure to be addressed by an efficient software development process together with an industrialization of the development. Industrialization requires defining the deliverables. Therefore, our data migration architecture extends the ETL process by migration objectives to be reached in each step. Industrialization also means standardizing the implementation, e.g. with patterns. We present data migration patterns describing the typical transformations found in the data migration application domain. Finally, testing is an important issue because testcase based testing cannot guarantee that not a single customer gets lost. Reconciliation can do so by checking whether each object in the old and new system has a counterpart in the other system.

show abstract

“…Data transformation is an old problem and the idea of using a query language to specify such transformations has been proposed back in the 1970's with two prototypes, Convert [SHL75] [CG04], have proposed operators for data transformation and cleaning purposes. Potter's Wheel fold operator is capable of producing several output tuples for each input tuple.…”

Section: Related Workmentioning

confidence: 99%

“…Our experience with Ajax [GFSS00] data cleaning tool and with Data Fusion [CG04] legacy-data migration tool has shown that in the context of data transformation, there is a considerable amount of data transformations that require one-to-many mappings. In fact, as recognized in [Gal01], an important class of data transformations require the inverse operation of the SQL group by/aggregates primitive that, for each input tuple, has the ability to produce several output tuples.…”

Section: Introductionmentioning

confidence: 99%

One-to-many data transformations through data mappers

Carreira¹,

Galhardas²,

Lopes³

et al. 2007

Data & Knowledge Engineering

Self Cite

View full text Add to dashboard Cite

The optimization capabilities of RDBMSs are turning them attractive for executing data transformations. However, despite the fact that many useful data transformations can be expressed as relational queries, an important class of data transformations that produce several output tuples for a single input tuple cannot be expressed in that way.To overcome this limitation, we propose to extend Relational Algebra with a new operator named data mapper. In this paper, we formalize the data mapper operator and investigate some of its properties. We then propose a set of algebraic rewriting rules that enable the logical optimization of expressions with mappers and prove their correctness. Finally, we validate experimentally the proposed optimizations and identify the key factors that influence the optimization gains.

show abstract

Efficient development of data migration transformations

Cited by 15 publications

References 6 publications

Data Mapper: An Operator for Expressing One-to-Many Data Transformations

Data Mapper: An Operator for Expressing One-to-Many Data Transformations

Towards the Industrialization of Data Migration: Concepts and Patterns for Standard Software Implementation Projects

One-to-many data transformations through data mappers

Contact Info

Product

Resources

About