Data location-aware job scheduling in the grid. Application to the GridWay metascheduler

Abstract

Abstract. Grid infrastructures constitute nowadays the core of the computing facilities of the biggest LHC experiments. These experiments produce and manage petabytes of data per year and run thousands of computing jobs every day to process that data. It is the duty of metaschedulers to allocate the tasks to the most appropriate resources at the proper time.Our work reviews the policies that have been proposed for the scheduling of grid jobs in the context of very data-intensive applications. We indicate some of the practical problems that such models will face and describe what we consider essential characteristics of an optimum scheduling system: aim to minimise not only job turnaround time but also data replication, flexibility to support different virtual organisation requirements and capability to coordinate the tasks of data placement and job allocation while keeping their execution decoupled.These ideas have guided the development of an enhanced prototype for GridWay, a general purpose metascheduler, part of the Globus Toolkit and member of the EGEE's RESPECT program. Current GridWay's scheduling algorithm is unaware of data location. Our prototype makes it possible for job requests to set data needs not only as absolute requirements but also as functions for resource ranking. As our tests show, this makes it more flexible than currently used resource brokers to implement different data-aware scheduling algorithms.

CMS Workflow Execution Using Intelligent Job Scheduling and Data Access Strategies

“…However, this approach is dependent upon the AlChemist framework and the WSPeer API to create data cache overlays on dedicated data nodes, whereas we have proposed to create data cache on every worker node inside a cluster to optimally use the available resources in the Grid infrastructures. There have been various approaches [14] to schedule job taking into account the location of the data needed by a job to minimize the data transfer, and thus the job execution time.…”

Αποτίμηση Της Απόδοσης Πολιτικών Χρονοδρομολόγησης Παράλληλων Εργασιών Σε Συστήματα Πλέγματος Με Μοντέλα Προσομοίωσης

“…Χρονοδρομολόγηση παράλληλων εργασιών με περιορισμούς προτεραιότητας 106 αυτού του είδους παράλληλων εφαρμογών μπορεί να αφορά τη χρήση πολιτικών χρονοδρομολόγησης τύπου ομάδας[72],[73] και επιλογή των πόρων με επίγνωση της θέσης τους (location-aware assignment)[74]. Η χρονοδρομολόγηση μιας παράλληλης Χρονοδρομολόγηση παράλληλων εργασιών με περιορισμούς προτεραιότητας 107 και περιορισμούς μεταξύ διεργασιών και πληροφοριών λανθάνουσας μνήμης (cacheaware schedulers) συνήθως δρομολογούν διεργασίες σε επεξεργαστές που έχουν πρόσφατα χρησιμοποιηθεί καθώς είναι πολύ πιθανό η λανθάνουσα μνήμη του επεξεργαστή να περιέχει χρήσιμες πληροφορίες για την επόμενη διεργασία[80].…”