Managing Cloud networking costs for data-intensive applications by provisioning dedicated network links

Abstract

Many scientific high-throughput applications can benefit from the elastic nature of Cloud resources, especially when there is a need to reduce time to completion. Cost considerations are usually a major issue in such endeavors, with networking often a major component; for data-intensive applications, egress networking costs can exceed the compute costs. Dedicated network links provide a way to lower the networking costs, but they do add complexity. In this paper we provide a description of a 100 fp32 PFLOPS Cloud burst in support of IceCube production compute, that used Internet2 Cloud Connect service to provision several logically-dedicated network links from the three major Cloud providers, namely Amazon Web Services, Microsoft Azure and Google Cloud Platform, that in aggregate enabled approximately 100 Gbps egress capability to on-prem storage. It provides technical details about the provisioning process, the benefits and limitations of such a setup and an analysis of the costs incurred. CCS CONCEPTS• Networks → Network performance evaluation; Network measurement; • Applied computing → Physical sciences and engineering; Astronomy.

Ultra-High-Energy Cosmic Rays: The Intersection of the Cosmic and Energy Frontiers

“…As deep learning relies on the availability of GPU resources that are not widely available in most university computing clusters, the pay-per-use of commercial cloud computing centers are a possible solution for the near future. The utility of cloud resources in astroparticle physics has already been explored by the IceCube collaboration for simulation to analyze the performance, usability, and running costs [876][877][878].…”