New YARN Non-Exclusive Resource Management Scheme through Opportunistic Idle Resource Assignment

Abstract

managing resources and improving throughput in a large-scale cluster has become a crucial problem with the explosion of data processing applications in recent years. Hadoop YARN and Mesos, as two universal resource management platforms, have been widely adopted in the commodity cluster for co-deploying multiple data processing frameworks, such as Hadoop MapReduce and Apache Spark. However, in the existing resource management, a certain amount of resources are exclusively allocated to a running task and can only be reassigned after that task is completed. This exclusive mode unfortunately leads to a potential problem that may under-utilize the cluster resources and degrade system performance. To address this issue, we propose a novel opportunistic and efficient resource allocation scheme, named OPERA, which breaks the barriers among the encapsulated resource containers by leveraging the knowledge of actual runtime resource utilizations to reassign opportunistic available resources to the pending tasks. OPERA avoids incurring severe performance interference to active tasks by further using two approaches to efficiently balances the starvations of reserved tasks and normal queued tasks. We implement and evaluate OPERA in Hadoop YARN v2.5. Our experimental results show that OPERA significantly reduces the average job execution time and increases the resource (CPU and memory) utilizations.

Memory aware optimized Hadoop MapReduce model in cloud computing environment

“…The linear regression approach is used as discussed in Institute of Electrical and Electronics Engineers, for data dependency, proposed model is designed to reduce the cost and total execution time for two distinctive applications i.e., text mining and iterative application [29]. Performance evaluation of both applications considering the existing Hadoop model is carried out in the next section.…”

“…In this section, the proposed method is evaluated considering the different parameters and constraints which is discussed later in the same section, Evaluation system parameter includes system configuration of Ubuntu 16 packed with dual-core 16 GB RAM. Furthermore, the Hadoop cluster along with two slaves and the master node is utilized to HDInsight Azure instance [29]. Performance evaluation is carried out on the standard dataset of Wikipedia, which varied up to 1024 MB, also further evaluation is carried out on complex sensor data up to 400 MB.…”

Task Offloading Scheme for Survivability Guarantee Based on Traffic Prediction in 6G Edge Networks

“…Due to the fact that traffic flow is a typical time series, we apply the Long-Short Term Memory (LSTM) algorithm to the traffic prediction model. LSTM is an improved recursive neural network algorithm that overcomes the problem of inaccurate learning of past information due to gradient vanishing in RNN [34]. By incorporating memory function and dynamically adjusting the correlation weight coefficients between sequences, LSTM achieves high-precision prediction of long and short time series.…”