Autonomous learning of load and traffic patterns to improve cluster utilization

Abstract

Adaptive clustering aims at improving cluster utilization for varying load and traffic patterns. Localitybased least-connection with replication (LBLCR) scheduling that comes with Linux is designed to help improve cluster utilization through adaptive clustering. A key issue with LBLCR, however, is that cluster performance depends much on a single threshold value that is used to determine adaptation. Once set, the threshold remains fixed, regardless of the load and traffic patterns. If a cluster of PCs is to adapt to different traffic patterns for high utilization, a good threshold has to be selected and used dynamically. We present in this paper an adaptive clustering framework that autonomously learns and adapts to different load and traffic patterns at runtime with no administrator intervention. The cluster is configured once and for all. As the patterns change, the cluster automatically expands/contracts to meet the changing demands. At the same time, the patterns are proactively learned so that when similar patterns emerge in the future, the cluster knows what to do to improve utilization. We have implemented this autonomous learning method and compared it with LBLCR using published Web traces. Experimental results indicate that our autonomous learning method produces high performance scalability and adaptability for different patterns. On the other hand LBLCR-based clustering suffers from performance scalability and adaptability for different traffic patterns since it is not designed to obtain good threshold values and use them at runtime.

A New MapReduce Framework Based on Virtual IP Mechanism and Load Balancing Strategy

“…Load balancing strategy [8] mainly includes RoundRobin Scheduling (RR), Weighted Round-Robin Scheduling (WRR) [9], Least-Connection Scheduling(LC) [10], Weighted Least-Connection Scheduling(WLC) [11], Locality-Based Least-Connection Scheduling(LBLC) [12], Locality-Based Least-Connection with Replication Schedul-ing(LBLCR), Destination Hashing Scheduling(DH), Source Hashing Scheduling and so on. Aiming at practical situation or experimental questions, algorithms can be restructured to optimize job assignment, such as appending dynamic detection based on the state of node, or adding algorithms based on some effective math conclusion into load balancing strategy.…”