Reducing Layered Database Applications to their Essence through Vertical Integration

Abstract

In the last decade, improvements on single-core performance of CPUs has stagnated. Consequently, methods for the development and optimization of software for these platforms have to be reconsidered. Software must be optimized such that the available single-core performance is exploited more effectively. This can be achieved by reducing the number of instructions that need to be executed. In this article, we show that layered database applications execute many redundant, nonessential, instructions that can be eliminated without affecting the course of execution and the output of the application. This elimination is performed using a vertical integration process which breaks down the different layers of layered database applications. By doing so, applications are being reduced to their essence, and as a consequence, transformations can be carried out that affect both the application code and the data access code which were not possible before. We show that this vertical integration process can be fully automated and, as such, be integrated in an operational workflow. Experimental evaluation of this approach shows that up to 95% of the instructions can be eliminated. The reduction of instructions leads to a more efficient use of the available hardware resources. This results in greatly improved performance of the application and a significant reduction in energy consumption.

A New Framework for Expressing, Parallelizing and Optimizing Big Data Applications

“…Rather than continuously executing this loop, it is only executed when a change is produced. To correct for the missing reinitialization, instead of initializing the values of PR at 1 |V | , they now need to be initialized at 1−d |V | . Unfortunately, while this specification is much less strict about order, it still contains a double iteration: first over vertices, and then over edges.…”

“…Note that more transformations, and many more implementations are possible. For a detailed description we refer to [11], and for the automatic generation of the associated data structures and the overall automation process we refer to [1], [14].…”

“…In order to do this, database queries were automatically transformed into explicit loop structures (forelem loops), which together with the wrapping C/C++ code allowed for integral optimization. This integral optimization led to significant performance improvements of database applications [1].…”

Redefining The Query Optimization Process

“…The optimization methodology that is proposed in this paper is part of a larger framework for the vertical integration of database applications [1]. Extensive vertical integration is not possible with traditional query optimization techniques, because when code is generated from query evaluation plans and combined with application code, further applicability of compiler transformations is obscured.…”