Performance Prediction for Data Transfers in LCLS Workflow

Abstract

In this work, we study the use of decision tree-based models to predict the transfer rates in different parts of the data pipeline that sends experiment data from Linac Coherent Light Source (LCLS) at SLAC National Accelerator Laboratory (SLAC) to National Energy Research Scientific Computing Center (NERSC). The system monitoring the data pipeline collects a number of characteristics such as the file size, source file system, start time and so on, all of which are known at the start of the file transfer. However, these static variables do not capture the dynamic information such as current state of the networking system. In this work, we explore a number of different ways to capture the state of the network and other dynamic information. We find that in addition to using static features, using these dynamic features can improve the transfer performance predictions by up to 10-15%. We additionally study a couple of different well-known decision-tree based models and find that Gradient-Tree Boosting algorithm performs better overall.

Performance Analysis and Optimization for Scientific Data Workloads

“…A number of works have addressed performance characterization of workflows running on NERSC resources from different aspects. Previous work [9] focuses on the data transfer part of the workflow and uses decision based techniques to predict the transfer rate of different file sizes from LCLS to NERSC. The decision is based on previously collected performance metrics such as transfer duration transfer start time, source file system, etc.…”

Evaluation of Deep Learning Models for Network Performance Prediction for Scientific Facilities

“…the average file size is greater than 40 GB, which takes around 55 seconds on average to move in, for a scientific application of Advanced Light Source (ALS) [1]. Linac Coherent Light Source (LCLS) at SLAC National Accelerator Laboratory produces terabytes of data for a single experiment, which is delivered to the National Energy Research Scientific Computing Center (NERSC) computing facility to process [5].…”

“…Another crucial dimension for large data transfers is the accurate network performance prediction, since estimating data transfer time is vital for workflow scheduling and resource allocation. In that regard, the connection log would be a helpful resource to infer the current and future network performance, such as for change point and anomaly detection [1,10] and for throughput and packet loss prediction [4,5].…”