Database Reverse Engineering

Profiling data to determine metadata about a given dataset is an important and frequent activity of any IT professional and researcher and is necessary for various use-cases. It encompasses a vast array of methods to examine datasets and produce metadata. Among the simpler results are statistics, such as the number of null values and distinct values in a column, its data type, or the most frequent patterns of its data values. Metadata that are more difficult to compute involve multiple columns, namely correlations, unique column combinations, functional dependencies, and inclusion dependencies. Further techniques detect conditional properties of the dataset at hand. This survey provides a classification of data profiling tasks and comprehensively reviews the state of the art for each class. In addition, we review data profiling tools and systems from research and industry. We conclude with an outlook on the future of data profiling beyond traditional profiling tasks and beyond relational databases.

show abstract

Profiling relational data: a survey

Abedjan¹,

2015

View full text Add to dashboard Cite

show abstract

Big Data Management: What to Keep from the Past to Face Future Challenges?

2017

View full text Add to dashboard Cite

The emergence of new hardware architectures, and the continuous production of data open new challenges for data management. It is no longer pertinent to reason with respect to a predefined set of resources (i.e., computing, storage and main memory). Instead, it is necessary to design data processing algorithms and processes considering unlimited resources via the ''pay-as-you-go'' model. According to this model, resources provision must consider the economic cost of the processes versus the use and parallel exploitation of available computing resources. In consequence, new methodologies, algorithms and tools for querying, deploying and programming data management functions have to be provided in scalable and elastic architectures that can cope with the characteristics of Big Data aware systems (intelligent systems, decision making, virtual environments, smart cities, drug personalization). These functions, must respect QoS properties (e.g., security, reliability, fault tolerance, dynamic evolution and adaptability) and behavior properties (e.g., transactional execution) according to application requirements. Mature and novel system architectures propose models and mechanisms for adding these properties to new efficient data management and processing functions delivered as services. This paper gives an overview of the different architectures in which efficient data management functions can be delivered for addressing Big Data processing challenges.

show abstract

Integrating legacy system into big data solutions: Time to make the change

Jha

O’Brien

et al. 2014

Asia-Pacific World Congress on Computer Science and Engineering

View full text Add to dashboard Cite

Storing, analyzing and accessing data is a growing problem for organizations. Competitive pressures and new regulations are requiring organizations to efficiently handle increasing volumes and varieties of data, but this doesn't come cheap. And as the demands of Big Data exceed the constraints of traditional relational databases, evaluating legacy infrastructure and assessing new technology has become a necessity for most organizations, not only to gain competitive advantage, but also for compliance purposes. The challenge is how well the organization's legacy infrastructure integrates Big Data. It is without a doubt that one way or another Big Data must be accommodated by legacy systems. Legacy systems contain significant and invaluable business logic of the organization. Organizations cannot afford to throw away or replace this business logic. These legacy systems are assets of the organization. These invaluable assets of encoded "business logic" represent many years of coding, development, real-life experiences, enhancements, modifications, debugging etc. Most of the legacy systems were developed without the process models or data models-now needed to support and integrate Big Data. To integrate Big Data into legacy system, modernization of legacy system is required. There are many approaches for modernization of legacy systems but none of them are focused on integrating Big Data into legacy systems. Legacy systems hold valuable data too important to be lost in the process of modernization. However, addressing the issues and scope related to incorporating Big Data with legacy systems allows mature legacy systems to become part of groundswell changes. There are many areas unaddressed about integration of Big Data into legacy systems. Incorporating data from new sources, specifically "live" sources, into existing legacy systems is a technical challenge. Moreover, the sheer volume of Big Data can be daunting. Our paper presents the scope of integrating Big Data into modernization of legacy systems.

show abstract

Database Reverse Engineering

Cited by 9 publications

References 52 publications

Profiling relational data: a survey

Profiling relational data: a survey

Big Data Management: What to Keep from the Past to Face Future Challenges?

Integrating legacy system into big data solutions: Time to make the change

Contact Info

Product

Resources

About