Multi-resource Low-latency Cluster Scheduling without Execution Time Estimation

Yabuuchi, Hidehito; Shinagawa, Takahiro

doi:10.1109/ccgrid49817.2020.00-62

Cited by 1 publication

(1 citation statement)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, Stratus is not DAG-aware and it only optimizes cost while AGORA can optimize for different objectives. Apart from the aforementioned offline schedulers that use workload runtime estimation or prior knowledge to achieve near-optimal results, online schedulers [3,15,43,44] require no prior task runtime information and adaptively schedule tasks during runtime. These approaches are orthogonal and could be applied as complementary to AGORA.…”

Section: Job Schedulingmentioning

confidence: 99%

Global Optimization of Data Pipelines in Heterogeneous Cloud Environments

Lin¹,

Xu²,

Bramhavar³

et al. 2022

Preprint

View full text Add to dashboard Cite

Modern production data processing and machine learning pipelines on the cloud are critical components for many cloudbased companies. These pipelines are typically composed of complex workflows represented by directed acyclic graphs (DAGs). Cloud environments are attractive to these workflows due to the wide range of choice with heterogeneous instances and prices that can provide the flexibility for different cost-performance needs. However, this flexibility also leads to the complexity of selecting the right resource configuration (e.g., instance type, resource demands) for each task in the DAG, while simultaneously scheduling the tasks with the selected resources to reach the optimal end-to-end performance and cost. These two decisions are often codependent resulting in an NP-hard scheduling optimization bottleneck. Existing solutions only focus solely on either problem and ignore the co-effect on the end-to-end optimum. We propose AGORA, a scheduler that considers both task-level resource allocation and execution for DAG workflows as a whole in heterogeneous cloud environments. AGORA first (1) studies the characteristics of the tasks from prior runs and gives predictions on resource configurations, and (2) automatically finds the best configuration with its corresponding schedules for the entire workflow with a costperformance objective. We evaluate AGORA in a heterogeneous Amazon Web Services (AWS) cloud environment with multi-tenant workflows served by Airflow and demonstrate a performance improvement up to 45% and cost reduction up to 77% compared to state-of-the-art schedulers. In addition, we apply AGORA to a real-world production trace from Alibaba and show cost reduction of 65% and DAG completion time reduction of 57%. Index Analysis SentimentAnalysis Airline Delay Movie Rec.

show abstract

Section: Job Schedulingmentioning

confidence: 99%