Towards a Novel Framework for Automatic Big Data Detection

Abstract

Big data is a "relative" concept. It is the combination of data, application, and platform properties. Recently, big data specific technologies have emerged, including software frameworks, databases, hardware accelerators, storage technologies, etc. However, the automatic selection of these solutions for big data computations remains a non-trivial task. Presently, the big data tools are selected by analyzing the problem manually, or by using several performance prediction techniques. The manual identification is based on the data properties only, whereas the performance predictors only estimate basic execution metrics without linking them with big data (3Vs) thresholds. Hence, both ways of identification are mostly incorrect, which can lead to inefficient use of 3Vs optimizations, resulting into global inefficiency, reduced system performance, increasing power consumption, requiring greater effort on the part of the programming team, and misallocation of the hardware resources required for the task. In this regard, a novel framework has been proposed for automatic detection of 3Vs (Volume, Velocity, Variety) of big data, using machine learning. The detection is done through static code features, data, and platform properties, leading to relevant tool selection, and code generation, with minimal overheads, lesser programmer interventions, higher usability, and portability. Instead of handling each application with big data specialized solutions, or manually identifying the 3Vs, the framework can automatically detect and link the 3Vs to the relevant optimizations. Several standard applications have been tested using the proposed framework. In the case of volume, the average detection accuracy is up to 97.8% for seen and 95.9% for unseen applications. In the case of velocity, the average detection accuracy is up to 97.3% for seen and 92.6% for unseen applications. There is no margin of error in variety detection, as it has straightforward computations without any predictions. Furthermore, an airline recommendation system case study strengthens the effectiveness of the proposed approach.

Toward a novel engine for compiler optimization space exploration of big data workloads

“…[2][3][4][5][6][7][8]36 The 3Vs (volume, velocity, variety) appear to be the determining characteristics of big data and are enough to model data processing challenge. [8][9][10]37,38 Volume is indicated by larger data size, velocityemerges when the data is arriving at a faster rate, and due to heterogeneous data formats variety is true. Due to the continuous generation of large size and multiple formats data, big data applications require fast processing at the same time in order to meet the deadlines.…”

“…In this manner, the processing task can be completed before the arrival of the new sample. 8,44 As indicated by Equation 1, process execution time P(T S,F ) exceeds the product of the total number of samples and data source sampling period (T s ). Where S and F indicate that the processing is done with large data sizes and heterogeneous formats.…”

“…In this manner, the execution time is required to be reduced for meeting the processing deadlines. 8 Big Data = true, if P(T S,F ) > Number of samples * P(T s ).…”

“…Such a huge amount of real-time heterogeneous data which is unable to get handled by conventional hardware or software solutions is termed big data. [2][3][4][5][6][7][8] The 3Vs (volume, velocity, variety) appear to be the defining characteristics of big data for modeling data processing challenge. 9,10 Currently, the big data problem has been tackled via specialized domain-specific languages, parallel programming models, in-memory databases, and third-party libraries.…”

“…Similarly, a team of trained personnel is deputed for processing big data problems using specialized tools as well as administrating the data centres. 8 These techniques are successful in concealing the low-level details by producing very high-level code, which is easier to understand, use, write, and maintain. However, this abstraction is intended only for the end-user, as internally the high-level code is passed through the conventional compilation cycle for generating hardware operable code.…”