Lessons Learned from a Decade of Providing Interactive, On-Demand High Performance Computing to Scientists and Engineers

Abstract

For decades, the use of HPC systems was limited to those in the physical sciences who had mastered their domain in conjunction with a deep understanding of HPC architectures and algorithms. During these same decades, consumer computing device advances produced tablets and smartphones that allow millions of children to interactively develop and share code projects across the globe. As the HPC community faces the challenges associated with guiding researchers from disciplines using high productivity interactive tools to effective use of HPC systems, it seems appropriate to revisit the assumptions surrounding the necessary skills required for access to large computational systems. For over a decade, MIT Lincoln Laboratory has been supporting interactive, ondemand high performance computing by seamlessly integrating familiar high productivity tools to provide users with an increased number of design turns, rapid prototyping capability, and faster time to insight. In this paper, we discuss the lessons learned while supporting interactive, on-demand high performance computing from the perspectives of the users and the team supporting the users and the system. Building on these lessons, we present an overview of current needs and the technical solutions we are building to lower the barrier to entry for new users from the humanities, social, and biological sciences.

LLload: Simplifying Real-Time Job Monitoring for HPC Users

“…The LLSC has a long history of lowering the barrier to entry for supercomputing practitioners by creating an environment where researchers can focus on short term mission driven applications [9]. As it became apparent that researchers needed a simplified, intuitive approach for job monitoring, the team began to design a tool that would…”

An Educational and Training Perspective on Integrating Hybrid Technologies with HPC Systems for Solving Real-World Commercial Problems

“…• Introductory and Learner level: Michael Nielsen's book [25] and tutorials [17,18,23,24] are one of the most welcoming when it comes to introduction for QC. As previously mentioned by [20][21][22], games and interactive environments have significant importance in education and training in HPC, and there are also QC-related games available [7,32]. There are several very well-prepared materials such as Qiskit textbook [8,31] Tensorflow Quantum [10], Cirq [13], Pennylane Tutorials [9], however, they are tailored for specific SDKs and hardware.…”

Benchmarking network fabrics for data distributed training of deep neural networks

“…For over a decade, the MIT Lincoln Laboratory Supercomputer Center (LLSC) has enabled interactive, on-demand high performance computing [14], [15] by integrating highproductivity tools with traditional HPC systems to enable the prototyping and development of a wide range of scientific applications. As machine learning (ML) workflows have occupied an increasingly large proportion of the types of jobs running on the system, we have developed new tools that support interactive web-based environments [16] such as Jupyter notebooks for prototyping machine learning applications.…”