Understanding the Workload Characteristics in Alibaba: A View from Directed Acyclic Graph Analysis

Lu, Chengzhi; Chen, Wenyan; Ye, Kejiang; Xu, Cheng‐Zhong

doi:10.1109/hpbdis49115.2020.9130578

Cited by 18 publications

(13 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Papers [4,6] only focus on the abnormal resource consumption event of the job. Paper [5] only focus on job scheduling failure events, papers [7][8][9][10] only focus on the abnormal consumption of container resources, and paper [11] only focus on instance authorization failure and Instantiate abnormal event.…”

Section: Cloud Service Eventsmentioning

confidence: 99%

See 1 more Smart Citation

A Survey of Cloud Service Events and Their Connections

Hu¹,

Zhang²,

Qin³

et al. 2022

Natural Language Processing (NLP) Trends

View full text Add to dashboard Cite

Any unexpected service interruption or failure may cause customer dissatisfaction or economic losses. To distinguish the rights and interests or security disputes between cloud service providers and customers, explore the essence and rules of cloud service events and their various connections, such as: Normal contact of service scheduling, normal contact of service dependence, abnormal contact of resource competition, abnormal contact of service delay, abnormal contact of service dependence, etc., as well as their rules in time, resources, scheduling and other aspects, and the form of the rules; The purpose is to provide the above abnormal connections, as well as the rule and presentation form in terms of time, resources and load, for the study of violation determination and failure tracing in the cloud service accountability mechanism.

show abstract

Section: Cloud Service Eventsmentioning

confidence: 99%

“…According to the papers [2][3][4][5][6][7][8][9][10][11][27][28][29][30][31], it is summarized that there are normal events, unknown events and abnormal events in cloud service events, this article focuses on abnormal events. The cloud service events classification diagram is shown in Figure 1.…”

Section: Cloud Service Events Classificationmentioning

confidence: 99%

A Survey of Cloud Service Events and Their Connections

Hu¹,

Zhang²,

Qin³

et al. 2022

Natural Language Processing (NLP) Trends

View full text Add to dashboard Cite

show abstract

“…We evaluate both the end-to-end runtime and cost of each DAG to demonstrate the efficacy of AGORA. Finally, we also apply AGORA on a real-world large-scale trace from Alibaba [29] to evaluate how AGORA would perform in a multi-DAG production environment.…”

Section: Dag1 Dag2mentioning

confidence: 99%

“…However, we are also interested in how AGORA scales and performs in real-world production environments. To this end, we use a real-world large-scale production trace from Alibaba [29] and apply AGORA with simulation. 5.5.1 Methodology.…”

Section: Macro-benchmarkmentioning

confidence: 99%

See 1 more Smart Citation

Global Optimization of Data Pipelines in Heterogeneous Cloud Environments

Lin¹,

Xu²,

Bramhavar³

et al. 2022

Preprint

View full text Add to dashboard Cite

Modern production data processing and machine learning pipelines on the cloud are critical components for many cloudbased companies. These pipelines are typically composed of complex workflows represented by directed acyclic graphs (DAGs). Cloud environments are attractive to these workflows due to the wide range of choice with heterogeneous instances and prices that can provide the flexibility for different cost-performance needs. However, this flexibility also leads to the complexity of selecting the right resource configuration (e.g., instance type, resource demands) for each task in the DAG, while simultaneously scheduling the tasks with the selected resources to reach the optimal end-to-end performance and cost. These two decisions are often codependent resulting in an NP-hard scheduling optimization bottleneck. Existing solutions only focus solely on either problem and ignore the co-effect on the end-to-end optimum. We propose AGORA, a scheduler that considers both task-level resource allocation and execution for DAG workflows as a whole in heterogeneous cloud environments. AGORA first (1) studies the characteristics of the tasks from prior runs and gives predictions on resource configurations, and (2) automatically finds the best configuration with its corresponding schedules for the entire workflow with a costperformance objective. We evaluate AGORA in a heterogeneous Amazon Web Services (AWS) cloud environment with multi-tenant workflows served by Airflow and demonstrate a performance improvement up to 45% and cost reduction up to 77% compared to state-of-the-art schedulers. In addition, we apply AGORA to a real-world production trace from Alibaba and show cost reduction of 65% and DAG completion time reduction of 57%. Index Analysis SentimentAnalysis Airline Delay Movie Rec.

show abstract

BBServerless: A Bursty Traffic Benchmark for Serverless

Lin

et al. 2022

Cloud Computing – CLOUD 2021

View full text Add to dashboard Cite

Understanding the Workload Characteristics in Alibaba: A View from Directed Acyclic Graph Analysis

Cited by 18 publications

References 16 publications

A Survey of Cloud Service Events and Their Connections

A Survey of Cloud Service Events and Their Connections

Global Optimization of Data Pipelines in Heterogeneous Cloud Environments

BBServerless: A Bursty Traffic Benchmark for Serverless

Contact Info

Product

Resources

About