Towards the Industrialization of Data Migration: Concepts and Patterns for Standard Software Implementation Projects

Abstract

Abstract. When a bank replaces its core-banking information system, the bank must migrate data like accounts from the old into the new system. Migrating data is necessary but not a catalyst for new business opportunities. The consequence is cost pressure to be addressed by an efficient software development process together with an industrialization of the development. Industrialization requires defining the deliverables. Therefore, our data migration architecture extends the ETL process by migration objectives to be reached in each step. Industrialization also means standardizing the implementation, e.g. with patterns. We present data migration patterns describing the typical transformations found in the data migration application domain. Finally, testing is an important issue because testcase based testing cannot guarantee that not a single customer gets lost. Reconciliation can do so by checking whether each object in the old and new system has a counterpart in the other system.

Testing & quality assurance in data migration projects

“…Above drivers entail the re-occurring replacement or consolidation of existing business applications as a type of maintenance. As a consequence, data migration projects represent an everlasting although infrequently performed discipline [11], [12]. Given their one-time event nature coupled with a permanent underestimation in size and complexity [13]- [15], existing know-how and experience is sparse and not subject to a diligent documentation process.…”

“…First, a data migration project requires additional time and effort which has to be performed on top of the ordinary IT tasks of plan, build, and run [12]. Unfortunately, this type of project is not business-enabling and the necessary time and costs are often underestimated [1], [11]. Secondly, it takes time and effort to build up internal competencies.…”

“…Data validation addresses four aspects: completeness, semantical correctness, consistency on the structure and the data level, and interoperability with other applications. Thus, a sophisticated combination of automated and manual comparisons has to be applied for validating the data and its structure [11]. The main challenge is to compare data of two-source and the target-databases, with respect to predefined comparison criteria [3], [25].…”

“…Such tests, typically named reconciliation [11], look at thousands of business objects demading for automating the comparison of the business objects in the source and the target database. Reconciliation scripts automatically map primary keys (respectively identifiers) of the source and target (reconciliation) to ensure completeness.…”

“…The imitating code should not be copied from the filter & transformation programs that are tested by the reconciliation. Instead, fresh code must be developed to not repeat past conceptual and implementation errors [9], [11]. Key deliverable of the completeness and type correspondence tests are, first, a list with all business objects missing in the target application and, second, a list with all business objects that are in the target application though they should not be.…”