NoSQL schema evolution and big data migration at scale

Klettke, Meike; Störl, Uta; Shenavai, Manuel; Scherzinger, Stefanie

doi:10.1109/bigdata.2016.7840924

Cited by 24 publications

(18 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the algorithm used by Baazizi et al ẇe used the kind equivalence relation, as we do with JSONoid. As noted previously, we are also able to produce a schema that contains the same information as the reduced structure identification graph of Klettke et al [21]. However, since their approach cannot scale beyond main memory, we exclude it from our comparative analysis.…”

Section: Runtimementioning

confidence: 99%

Jsonoid: Monoid-Based Enrichment for Configurable and Scalable Data-Driven Schema Discovery

Mior¹

2023

Preprint

View full text Add to dashboard Cite

Section: Runtimementioning

confidence: 99%

Jsonoid: Monoid-Based Enrichment for Configurable and Scalable Data-Driven Schema Discovery

Mior¹

2023

Preprint

View full text Add to dashboard Cite

“…Some approaches suggest lazy data migration for very large NoSQL databases, for instance, in Scherzinger et al [13], and in Saur et al [12], the overhead of lazy migration is discussed in terms of NoSQL databases. The foundations of different data migration strategies like incremental and predictive migration have been introduced in [6] and investigated in Klettke et al [4]. Although workload monitoring can be applied for different tasks like automated database tuning [11], we are not aware of any approaches for automated selection and adaptation of these data migration approaches as has been presented in this paper.…”

Section: Related Workmentioning

confidence: 99%

Self-adapting data migration in the context of schema evolution in NoSQL databases

Hillenbrand

Störl

Nabiyev

et al. 2021

Distrib Parallel Databases

Self Cite

View full text Add to dashboard Cite

When NoSQL database systems are used in an agile software development setting, data model changes occur frequently and thus, data is routinely stored in different versions. The management of versioned data leads to an overhead potentially impeding the software development. Several data migration strategies exist that handle legacy data differently during data accesses, each of which can be characterized by certain advantages and disadvantages. Depending on the requirements for the software application, we evaluate and compare different migration strategies through metrics like migration costs and latency as well as precision and recall. Ideally, exactly that strategy should be selected whose characteristics fulfill service-level agreements and match the migration scenario, which depends on the query workload and the changes in the data model which imply an evolution of the database schema. In this paper, we present a methodology of self-adapting data migration, which automatically adjusts migration strategies and their parameters with respect to the migration scenario and service-level agreements, thereby contributing to the self-management of database systems and supporting agile development.

show abstract

“…The estimates of [15] are based on discrete-event simulation using workload and structure models taken from logs, as well as the schema of the tobe-migrated database, whereas we investigate a range of migration scenario characteristics in a Monte Carlo approach. Since the traditional approach of eager migration can become very expensiveespecially in a cloud environment [15,22]-other approaches to data migration such as lazy [27,35] and proactive [22] approaches have been proposed. To our knowledge, there is no related study on the effects of the various migration strategies, comparable to ours in its systematics.…”

Section: Related Workmentioning

confidence: 99%

“…If, on the other hand, saving costs is the most important criterion, then a lazy migration strategy should be applied which minimizes migration costs, as data remains unchanged in the event of a release. In this case, if a legacy entity is accessed, it is migrated individually and on-the-fly, then being in congruence with the latest schema version, yet introducing a considerable runtime overhead [22,27,35]. These two metrics, migration costs and data access latency, are in fact competitors in a tradeoff, which is schematically depicted in Figure 1.…”

Section: Introductionmentioning

confidence: 99%

MigCast in Monte Carlo: The Impact of Data Model Evolution in NoSQL Databases

Hillenbrand,

Störl,

Nabiyev

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

During the development of NoSQL-backed software, the data model evolves naturally alongside the application code. Especially in agile development, new application releases are deployed frequently causing schema changes. Eventually, decisions have to be made regarding the migration of versioned legacy data which is persisted in the cloud-hosted production database. We solve this schema evolution problem and present the results of near-exhaustive calculations by means of which software project stakeholders can manage the operative costs for data model evolution and adapt their software release strategy accordingly in order to comply with service-level agreements regarding the competing metrics of migration costs and latency. We clarify conclusively how data model evolution in NoSQL databases impacts the metrics while taking all relevant characteristics of migration scenarios into account. As calculating all possible combinatorics in the search space of migration scenarios would by far exceed computational means, we used a probabilistic Monte Carlo method of repeated sampling, serving as a well-established means to bring the complexity of data model evolution under control. Our experiments show the qualitative and quantitative impact on the performance of migration strategies with respect to intensity and distribution of data entity accesses, the kinds of schema changes, and the characteristics of the underlying data model.

show abstract

NoSQL schema evolution and big data migration at scale

Cited by 24 publications

References 7 publications

Jsonoid: Monoid-Based Enrichment for Configurable and Scalable Data-Driven Schema Discovery

Jsonoid: Monoid-Based Enrichment for Configurable and Scalable Data-Driven Schema Discovery

Self-adapting data migration in the context of schema evolution in NoSQL databases

MigCast in Monte Carlo: The Impact of Data Model Evolution in NoSQL Databases

Contact Info

Product

Resources

About