A comparison of general-purpose distributed systems for data processing

Deterministic, elastic and real - time processing in the big data era

“…An important observation is that the current state-of-the art big data frameworks differ significantly on the way they exploit the available processing resources, the configuration parameters that can be tuned (e.g., the amount of memory an application will reserve) and the policies they follow in decomposing the processing into multiple tasks. It has been documented that notable discrepencies in terms of performance can be observed between implementations of the same application in different frameworks [121]. We verify this observation in our local 8 node cluster (i.e., 7 workers and 1 master node).…”

“…We use Spark 2.0.2, Flink 1.4.0 and TensorFlow 1.3.0. For the evaluation of Orion we consider eight big data analysis applications used in similar settings [121,137] (see also Table 6.2). Applications are submitted in Orion in a round-robin fashion and the applications' submission time follows a Poisson distribution.…”

“…As can be observed in Figures 6.1a, 6.1b, 6.1c and 6.1d, no framework outperforms all others for the applications under consideration. In general, Spark is more efficient than the others for most applications as it was also reported in [121] mainly due to its popularity. However, Flink and TensorFlow outperform Spark in some cases.…”