A progress and profile-driven cloud-VM for resource-efficiency and fairness in e-science environments

Abstract

Cloud platforms are becoming more prevalent in e-Science domains, also by encompassing new and existing Grid infrastructures into private, hybrid and federated clouds. Clouds are inherently multi-tenant as they run workloads from multiple users. Resources can be initially allocated statically, as for job scheduling in Grids previously, but they can also be changed elastically at runtime to meet the application effective needs.When allocation needs to be changed, and resources are scarce, determining from which tenants resources must be taken to impact performance the least is a non-trivial and often deemed intractable problem, when outside the realm of batch scheduling and full prior information on resource requirements for each task, job, or VM instance.In this paper we present a Java-based platform for cloud environments that is able to : i) monitor application progress with different levels-of-detail and allowing full applications transparency, ii) account and restrict resource consumption, such as CPU and memory, by applications, and iii) A clusterwide and decentralized algorithm that, based on the progress of different workloads, can redistribute resources among different JVM instances. Evaluation shows it is able to improve resource-efficiency and fairness across e-Science private cloud infrastructures, by managing and migrating resources according to the previous criteria, driven by a number of novel proposed metrics inspired in Economics.