TOPOSCH: Latency-Aware Scheduling Based on Critical Path Analysis on Shared YARN Clusters

Hu, Chunming; Zhu, Jianyong; Yang, Renyu; Peng, Hao; Wo, Tianyu; Xue, Shiqing; Yu, Xiao‐Qiang; Xu, Jie; Ranjan, Rajiv

doi:10.1109/cloud49709.2020.00091

Cited by 8 publications

(11 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, unawareness of application-level QoS at runtime could lead to host resource uneven or over-saturation-intrusive applications consume too many resources-making neighbor protected workloads experience performance outliers. 14,16 As shown in Figure 1A,B, we observed the JCT of the Spark Kmeans job co-located with stream 17 and the JCT of the Mapreduce Terasort job co-located with fio. 18 With the increase in the concurrency of the co-located intrusive workloads, the JCT of spark and Mapreduce jobs continue to grow, and the growth has gone from initially flat to relatively sharp as intrusive workloads steal more and more resources.…”

Section: Performance Interferencementioning

confidence: 73%

“…Many studies also improve the effectiveness of performance isolation through intrusive instrumentation and modification at application-level. 1,14 Unfortunately, existing research is not suitable for performance isolation in large-scale shared clusters with scale-out workloads.…”

Section: Introductionmentioning

confidence: 99%

“…Research in Reference 10 restart multiple copies of stragglers detected belong to a scale-out job and select the earliest copy that finished. In our previous work, 14 we designed the latency-aware scheduling to guide the tasks placement for scale-out applications to reduce the performance interference caused by batch jobs. However, this method of hindsight caused a certain waste of resources, and the reappearance of stragglers cannot be avoided.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

ScaleReactor: A graceful performance isolation agent with interference detection and investigation for container‐based scale‐out workloads

Zhu

et al. 2021

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

Striking a balance between improved cluster utilization and guaranteed applicationQoS is a long-standing research problem in multi-tenants shared cluster. The typical solution is to detect performance degradation and investigate the root cause to conduct performance isolation. Existing efforts rely on lots of prior knowledge of applications and the assumption of interference-free workload placement is possible.Performance interference is usually mitigated through application-level approaches such as centralized rescheduling, which is usually an hindsight and a waste of resources.In this article, we present ScaleReactor, a graceful runtime agent on a per node basis that decouples the performance isolation from centralized resource management, and migrates the performance interference of scale-out workloads in container-based cluster using a lightweight black-box approach. ScaleReactor analyzes the degree of contention for multi-dimensional resources among co-located workloads to detect the performance degradation without additional prior information, and uses correlation analysis to locate the cause of contention, while isolating resources in a graceful manner to reduce system overhead and the performance degradation of intrusive workloads. Experiments have demonstrated that ScaleReactor effectively reduces the job completion time of scale-out applications in shared clusters, with the maximum value up to 36% and low system overhead against the existing isolation mechanism.

show abstract

Section: Performance Interferencementioning

confidence: 73%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

ScaleReactor: A graceful performance isolation agent with interference detection and investigation for container‐based scale‐out workloads

Zhu

et al. 2021

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

show abstract

“…Similarly, a work named ABS-YARN [16] uses a state-ofthe-art resource negotiator to quickly make decisions at the modeling level and reduce unneeded costs. Lastly in the topic of resource management, by harvesting run time latency, Toposch [17] can co-locate batches and microservices.…”

Section: Related Workmentioning

confidence: 99%

Optimizing Internal Overlaps by Self-Adjusting Resource Allocation in Multi-Stage Computing Systems

et al. 2021

View full text Add to dashboard Cite

With the rise of big data, more and more users will launch computing systems to process a large volume of data in various applications. A Scheduling algorithm is crucial to the performance of the processing platforms, especially when they are concurrently executing a batch of jobs. Such jobs usually represent multiple stages. Each stage produces the intermediate data which will be piped to the next stage for further processing. However, the scheduling problem in a big data computing system is different from the traditional multi-stage job scheduling problem as for any two consecutive stages, the later stage usually starts before the former stage is finished to "shuffle" the intermediate data. In this paper, we consider MapReduce/Hadoop as a representative computing system and develop a new strategy named OMO. A MapReduce job contains two consecutive phases: map and reduce. Our general target is to optimize the internal overlap between these two phases. There are two new techniques included in our solution, Lazy start of reduce tasks and Batch finish of map tasks, which aim to approach an effective alignment of the two phases based on the characteristics of the MapReduce process. OMO has been implemented on the Hadoop system with extensive experiments for performance evaluation. The results show that OMO's performance is superior in terms of total completion time (i.e., makespan) of a batch of jobs.

show abstract

“…Cluster-based method is a machine learning technique that involves grouping data points; the detailed description of the method is that given a set of data points, clustering algorithms can be used to divide each data point into a specific group [8]. In theory, data points in the same group should have similar properties, and data points in different groups should have highly different attributes.…”

Section: Introductionmentioning

confidence: 99%

An Anomaly Event Detection Method Based on GNN Algorithm for Multi-data Sources

Wang

et al. 2021

Proceedings of the 3rd ACM International Symposium on Blockchain and Secure Critical Infrastructure

View full text Add to dashboard Cite

Anomaly event detection is crucial for critical infrastructure security(transportation system, social-ecological sector, insurance service, government sector etc.) due to its ability to reveal and address the potential cyber-threats in advance by analysing the data(messages, microblogs, logs etc.) from digital systems and networks. However, the convenience and applicability of smart devices and the maturity of connected technology make the social anomaly events data multi-source and dynamic, which result in the inadaptability for multi-source data detection and thus affect the critical infrastructure security. To effectively address the proposed problems, in this paper, we design a novel anomaly detection method based on multi-source data. First, we leverage spectral clustering algorithm for feature extraction and fusion of multiple data sources. Second, by harnessing the power of deep graph neural network(Deep-GNN), we perform a fine-gained anomaly social event detection, revealing the threatening events and guarantee the critical infrastructure security. Experimental results demonstrate that our framework outperforms other baseline anomaly event detection methods and shows high tracking accuracy, strong robustness and stability. CCS CONCEPTS• Security and privacy → Domain-specific security and privacy architectures; • Computing methodologies → Machine learning.

show abstract

TOPOSCH: Latency-Aware Scheduling Based on Critical Path Analysis on Shared YARN Clusters

Cited by 8 publications

References 26 publications

ScaleReactor: A graceful performance isolation agent with interference detection and investigation for container‐based scale‐out workloads

ScaleReactor: A graceful performance isolation agent with interference detection and investigation for container‐based scale‐out workloads

Optimizing Internal Overlaps by Self-Adjusting Resource Allocation in Multi-Stage Computing Systems

An Anomaly Event Detection Method Based on GNN Algorithm for Multi-data Sources

Contact Info

Product

Resources

About