Multi Attribute D-S Evidence Theory Based OCC for Shared-State Scheduling in Large Scale Cluster

Abstract

Abstract-With the growth of big data problems, nowadays the size of cloud-scale computing clusters is growing rapidly to run complicated parallel processing jobs. To full utilize cluster resources, the cluster management system is being challenged by the scaling cloud size and the often more complicated application requirements. Omega scheduling software provides a flexible and scalable shared-state scheduling architecture for large scale cluster scheduling. One of its key ideas is using an optimistic concurrency control (OCC) algorithm to let parallel schedulers concurrently make decisions. However, there are few studies exploring to extend OCC for a shared-state scheduling architecture. Furthermore, most of the traditional' shared-state scheduling architectures also use the same OCCs as Omega does. In this paper, we present a multi attribute Dempster-Shafer (D-S) evidence theory based OCC for shared-state scheduling. This OCC adapts the multi attribute D-S evidence theory to help making conflict decisions for some scheduling transactions. Experiments' results show that our method can obtain in some respects more optimized scheduling results compared to coarse-grained conflict detection of Omega.Index Terms-large scale cluster scheduling; multi attribute D-S evidence theory; optimistic concurrency control; Shared-state scheduling