Leveraging OpenStack and Ceph for a Controlled-Access Data Cloud

Abstract

While traditional HPC has and continues to satisfy most workflows, a new generation of researchers has emerged looking for sophisticated, scalable, on-demand, and self-service control of compute infrastructure in a cloud-like environment. Many also seek safe harbors to operate on or store sensitive and/or controlled-access data in a high capacity environment.To cater to these modern users, the Minnesota Supercomputing Institute designed and deployed Stratus, a locally-hosted cloud environment powered by the OpenStack platform, and backed by Ceph storage. The subscription-based service complements existing HPC systems by satisfying the following unmet needs of our users: a) on-demand availability of compute resources; b) long-running jobs (i.e., > 30 days); c) container-based computing with Docker; and d) adequate security controls to comply with controlled-access data requirements.This document provides an in-depth look at the design of Stratus with respect to security and compliance with the NIH's controlledaccess data policy. Emphasis is placed on lessons learned while integrating OpenStack and Ceph features into a so-called "walled garden", and how those technologies influenced the security design. Many features of Stratus, including tiered secure storage with the introduction of a controlled-access data "cache", fault-tolerant livemigrations, and fully integrated two-factor authentication, depend on recent OpenStack and Ceph features.

An adaptive read/write optimized algorithm for Ceph heterogeneous systems via performance prediction and multi-attribute decision making

“…Furthermore, we could develop more practical cloud service discovery mechanisms in cloud environments in the future [3,4]. Typical storage systems include GFS [5], Ceph [6], Azure Storage [7], Amazon S3 [8], Openstack [9], etc. These storage systems are linked to many common servers to provide storage services to the public.…”