MapReduce vs Non-MapReduce - Efficiency and Scalability in Big Data Computing

Abstract

MapReduce is a popular distributed computing paradigm for processing big data in a massively parallel fashion. However, when it is used to implement and run highly iterative algorithms for analyzing distributedly stored big data, the MapReduce paradigm loses its computing efficiency and data scalability due to the communication costs occurring in iterations of the algorithm over the entire dataset. Non-MapReduce is an alternative computing paradigm that removes the communication costs when executing iterative algorithms on big data that is stored using the random sample partition data model. In the Non-MapReduce paradigm, a set of random sample data blocks are selected and loaded into the memory of computing nodes. An iterative algorithm is dispatched to each computing node and executed on local data set independently and in parallel without communications among the nodes. Afterwards, the local results are transferred to the master node for computing the final result. In this paper, we propose the LOGO computing framework, a core technology for Non-MapReduce paradigm, and demonstrate its computing performance with widely used supervised learning, unsupervised learning, and pattern mining algorithms. The experiment results show that LOGO outperforms the state-of-the-art Spark by orders of magnitude in terms of the running time. LOGO is scalable to terabyte-scale data sets with high-quality results.Аннотация. MapReduce -это популярная парадигма распределенных вычислений для обработки больших данных в массово-параллельном режиме. Однако когда он используется для реализации и запуска высокоитеративных алгоритмов анализа распределенно хранящихся больших данных, парадигма MapReduce теряет свою вычислительную эффективность и масштабируемость данных из-за затрат на связь, возникающих при итерациях алгоритма по всему набору данных. Non-MapReduce -это альтернативная парадигма вычислений, которая устраняет затраты на связь при выполнении итерационных 244 Теория систем, алгебраическая биология, искусственный интеллект: математические основы и приложения