Optimization of tomographic reconstruction workflows on geographically distributed resources

Abstract

New technological advancements in synchrotron light sources enable data acquisitions at unprecedented levels. This emergent trend affects not only the size of the generated data but also the need for larger computational resources. Although beamline scientists and users have access to local computational resources, these are typically limited and can result in extended execution times. Applications that are based on iterative processing as in tomographic reconstruction methods require high-performance compute clusters for timely analysis of data. Here, time-sensitive analysis and processing of Advanced Photon Source data on geographically distributed resources are focused on. Two main challenges are considered: (i) modeling of the performance of tomographic reconstruction workflows and (ii) transparent execution of these workflows on distributed resources. For the former, three main stages are considered: (i) data transfer between storage and computational resources, (i) wait/queue time of reconstruction jobs at compute resources, and (iii) computation of reconstruction tasks. These performance models allow evaluation and estimation of the execution time of any given iterative tomographic reconstruction workflow that runs on geographically distributed resources. For the latter challenge, a workflow management system is built, which can automate the execution of workflows and minimize the user interaction with the underlying infrastructure. The system utilizes Globus to perform secure and efficient data transfer operations. The proposed models and the workflow management system are evaluated by using three high-performance computing and two storage resources, all of which are geographically distributed. Workflows were created with different computational requirements using two compute-intensive tomographic reconstruction algorithms. Experimental evaluation shows that the proposed models and system can be used for selecting the optimum resources, which in turn can provide up to 3.13Â speedup (on experimented resources). Moreover, the error rates of the models range between 2.1 and 23.3% (considering workflow execution times), where the accuracy of the model estimations increases with higher computational demands in reconstruction tasks.

Trace: a high-throughput tomographic reconstruction engine for large-scale datasets

“…Although these works show satisfactory reconstruction performance, most of them focus on improving the performance of a specific reconstruction algorithm with shared memory parallelization. In our work, we consider easing the implementation and parallelization of different reconstruction algorithms using a MapReduce-like middleware [ 6 , 12 , 13 ], and scale reconstruction operations to many compute nodes.…”

“…When the column sizes are doubled, however, the reconstruction times show an almost exponential increase. The main reason for this higher sensitivity to column sizes (i.e., x dimension) is the relationship between the number of variables in input dataset and output 3D image [ 13 ].…”

Deep Learning Accelerated Light Source Experiments

“…Advanced parallelization techniques, such as in-slice parallelization [8,9] and memory-centric [10] approaches, address the limitations of the naive approach. In-slice parallelization replicates sinogram and image among the processes and perform global reduction at the end of each iteration; therefore, the portions of the same sinogram can be reconstructed by multiple processes.…”

“…The sustained data consumption rate is measured by the number of projection processed per second. 9. 9.7 12.9 20.4 15.4 16.4 20.1 26.4 40.8 Glass Sustained Rate (p/s) 10.7 20.8 36.9 56.0 75.1 2.1 4.1 6.7 10.6 14.7…”

IDEAL: Images Across Domains, Experiments, Algorithms and Learning

“…There has been interest throughout the community in creating tools that can tackle scientific images. Other efforts besides ours include (1) tools at the ANL Advanced Photon Source, including Tomopy 11 for image reconstruction 12 and Midas 13 for analysis of grain interrelationships in crystalline materials; (2) work on the Materials Knowledge System (MKS), led by researchers at Georgia Tech, supports multi-scale materials science investigations using python packages that enable a range of functions, from synthetic data construction to spatial statistics; and (3) PyHST2, from the European Synchrotron Facility (ESRF), which exploits hybrid architectures using both central processing units (CPUs) and GPUs to deliver parallel processing techniques. While our multidisciplinary teams leverage some of these tools for the data reconstruction, IDEAL apps are focused on recognizing geometrical structures and measuring material deformation from 3D structured meshes, like image stacks, as the initial point.…”