Spark on entropy: A reliable &amp; efficient scheduler for low-latency parallel jobs in heterogeneous cloud

Chen, Huankai; Wang, Frank Z.

doi:10.1109/lcnw.2015.7365918

Cited by 14 publications

(6 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As shown through Formulas 1 and 2, these were then calculated, where j ∈ [0.1, 0.2, 0.3, 0.4, 0.5] represents the degree of complexity and i ∈ [5,6,7,8,9,10,11,12,13,14,15] represents the number of VMs.…”

Section: B Experiments Resultsmentioning

confidence: 99%

“…The reported scientific advances in both software platform development and Cloud Computing that enable fast data processing in the cloud is certainly a good news. Successful deployment of analysis engines such as Hive, Dremel, MapReduce, Spar and Impala has helped to run analysis jobs in short time across thousands of cloud resources [11]. However, adaptively scheduling groups of tasks based on dynamic changes in resource performance has been a challenge and remains unsolved.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

ComplexCloudSim: Towards Understanding Complexity in QoS-Aware Cloud Scheduling

Chen¹,

Wang²

2017

ijacsa

Self Cite

View full text Add to dashboard Cite

Abstract-The cloud is generally assumed to be homogeneous in most of the research efforts related to cloud resource management and the performance of cloud resource can be determined as it is predictable. However, a plethora of complexities are associated with cloud resources in the real world: dynamicity, heterogeneity and uncertainty. For heterogeneous cloud resources experiencing vast dynamic changes in performance, a critical role is played by the statistical characteristics of execution times, related to different cloud resources, to facilitate decision making in management. The cloud's performance can be considerably influenced by the differences between the estimated and actual execution times, which may affect the robustness of resource management systems.Limitation exists in the study of cloud resource management systems' complexities even though extensive research has been done on complexity issues in various fields from decision making in economics to computational biology. This paper concentrates on managing the research question regarding the complexity's role in QoS-aware cloud resource management systems. We present the ComplexCloudSim. Here, CloudSim, a popular simulation tool-kit, is extended through modelling of complexity factors in the cloud, including dynamic changes of run-time performance, resource heterogeneity, and task execution times' uncertainty. The effects of complexity on performance within cloud environments are examined by comparing four widely used heuristic cloud scheduling algorithms, given that the execution time information is inaccurate. Furthermore, a damage spreading analysis, one amongst the available complex system analysis methods, is applied to the system and simulations are run to reveal the system's sensitivity to initial conditions within specific parameter regions. Finally, how small of a damage can spread throughout the system within the region is discussed as well as research is done for the potential ways to avoid such chaotic behaviours and develop more robust systems.

show abstract

Section: B Experiments Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

ComplexCloudSim: Towards Understanding Complexity in QoS-Aware Cloud Scheduling

Chen¹,

Wang²

2017

ijacsa

Self Cite

View full text Add to dashboard Cite

show abstract

“…In this section, we will compare TOPE with the existing approach FIFO, SpreadOut and Non-SpreadOut [26], Fair Scheduler [27] and Naïve Bayes Scheduler [28] through the following aspects: bandwidth resource cost, load balancing effect and energy consumption.…”

Section: Methodsmentioning

confidence: 99%

“…if the episode ends at the state s j+1 ; r j + γ max a Q(s , , a , , ω , ), otherwise. (27) where Q(s , a', ω ) is the Q-value of the target network. The parameters of the convolutional neural network are updated according to the gradient descent method.…”

Section: B Virtual Link Mapping Schemementioning

confidence: 99%

A Novel Task Provisioning Approach Fusing Reinforcement Learning for Big Data

Cheng

2019

IEEE Access

View full text Add to dashboard Cite

The large-scale tasks processing for big data using cloud computing has become a hot research topic. Most of previous work on task processing is directly customized and achieved through existing methods. It may result in relatively more system response time, high algorithm complexity and resource waste, etc. Based on this argument, aiming at realizing overall load balancing, bandwidth cost minimization and energy conservation while satisfying resource requirements, a novel large-scale tasks processing approach called TOPE (Two-phase Optimization for Parallel Execution) is developed. The deep reinforcement learning model is designed for virtual link mapping decisions. We treat whole network as a multi-agent system and the whole process of selecting each node's next hop node is formalized via Markov decision process. We train the learning agent by deep neural network to store parameters of deep network model while approximating the value function, rather than tons of state-action values. The virtual node mapping is achieved by designed distributed multi-objective swarm intelligence to realize our two-phase optimization for task allocation in topology structure of Fat-tree. We provide experiments to show the ability of TOPE in analyzing task requests and infrastructure network. The superiority of TOPE for largescale tasks processing is convincingly demonstrated by comparing with state-of-the-art approaches in cloud environment.

show abstract

“…To evaluate the spread of the damages, we define damage D average (Difference of average workflow runtime R average ) and D std (Difference of workflow runtime Standard Deviation R std ) between two simulations results, which are calculated as shown in Formula 2 and 3, where i ∈ [5,6,7,8,9,10,11,12,13,14,15] As we can see from Figure 5, for number of VMs i < 10, the changes of D average for different degrees of complexity is relatively small, in this region, the damage is not spread and initial damage stays small.…”

Section: Order and Chaos In Complex Scheduling Systemmentioning

confidence: 99%

Complexity Reduction: Local Activity Ranking by Resource Entropy for QoS-Aware Cloud Scheduling

Chen¹,

Wang²,

Migliavacca³

et al. 2016

2016 IEEE International Conference on Services Computing (SCC)

Self Cite

View full text Add to dashboard Cite

The principle of local activity originated from electronic circuits, but can easily translate into other non-electrical homogeneous/heterogeneous media. Cloud resource is an example of a locally-active device, which is the origin of complexity in cloud scheduling system. However, most of the researchers implicitly assume the cloud resource to be locally passive when constructing new scheduling strategies. As a result, their research solutions perform poorly in the complex cloud environment. In this paper, we first study several complexity factors caused by the locally-active cloud resource. And then we extended the "Local Activity Principle" concept with a quantitative measurement based on Entropy Theory. Furthermore, we classify the scheduling system into "Order" or "Chaos" state with simulating complexity in the cloud. Finally, we propose a new approach to controlling the chaos based on resource's Local Activity Ranking for QoS-aware cloud scheduling and implement such idea in Spark. Experiments demonstrate that our approach outperforms the native Spark Fair Scheduler with server cost reduced by 23%, average response time improved by 15%-20% and standard deviation of response time minimized by 30%-45%.

show abstract

Spark on entropy: A reliable & efficient scheduler for low-latency parallel jobs in heterogeneous cloud

Cited by 14 publications

References 12 publications

ComplexCloudSim: Towards Understanding Complexity in QoS-Aware Cloud Scheduling

ComplexCloudSim: Towards Understanding Complexity in QoS-Aware Cloud Scheduling

A Novel Task Provisioning Approach Fusing Reinforcement Learning for Big Data

Complexity Reduction: Local Activity Ranking by Resource Entropy for QoS-Aware Cloud Scheduling

Contact Info

Product

Resources

About