Slurm Scheduling From Rules-Based Systems

Abstract

This paper explores how a rules based scheduling system can be integrated with a traditional workload manager such as slurm. This integration will be done with as minimal additional setup by a user as is feasible, while maintaining security of the network, as well as any machines involved. To this effect the processing component known as the Conductor in the rules-based scheduling system MEOW, has been extended with a remote option. This will enable MEOW to transmit jobs to a remote system, with the option of using slurm to orchestrate them. The new option is evaluated by comparing the execution time of using the remote solution without slurm, with that of existing components. Furthermore, the overhead of using various slurm methods for scheduling jobs with that of running the jobs remotely without using slurm is compared. It is shown that the remote solution adds a flat overhead to the execution time as both the number of jobs and size of the transmitted data increases, and that the overhead associated with using slurm on top of it is largely insignificant. This is considered to be an acceptable result given the test environment that was used.