H-Scheduler: Storage-Aware Task Scheduling for Heterogeneous-Storage Spark Clusters

Trident

“…H-Scheduler [31] is the only other storage-aware task scheduler designed to work over a tiered storage system such as HDFS (with tiering enabled). The key idea of H-Scheduler is to classify the tasks by both data locality and storage types, and redefine their scheduling priorities.…”

“…The key idea of H-Scheduler is to classify the tasks by both data locality and storage types, and redefine their scheduling priorities. Specifically, given available resources on some cluster node, schedule tasks based on the following priorities: local memory > local SSD > local HDD > remote HDD > remote SSD > remote memory [31]. The main issue with H-Scheduler and Quartet is that their heuristic methodology implements a best-effort approach that (in many cases) leads to sub-optimal or even poor task assignments, as we will see in Section 6.…”

“…Schedulers. In addition to our Trident Scheduler, we implemented two more schedulers from recent literature, namely H-Scheduler [31] and Quartet [14] (as described in Section 2), within the MapReduce Application Master and the Spark Driver. When running the Hadoop experiments, we used the Capacity Scheduler in YARN's Resource Manager, as was done in [14].…”

“…Hence, the execution times of a set of data-local tasks can vary significantly depending on the I/O performance of the storage media each task is reading from. In addition, as the bandwidth of the interconnection network keeps increasing (and headed towards In-finiBand), the execution time bottleneck of tasks is shifting towards the I/O performance of the storage devices [5,31].…”

“…In the presence of an in-memory caching tier, some systems like PACMan [6], BigSQL [16], and Quartet [14] will simply prioritize scheduling memory-local tasks over data-local tasks (as they assume that only two tiers exist). H-Scheduler [31] is perhaps the only other storage-aware scheduler implemented for Spark over tiered HDFS. H-Scheduler employs a heuristic algorithm for scheduling tasks to available resources that, unlike our approach, does not guarantee an optimal task assignment.…”