SciSpace: A scientific collaboration workspace for geo-distributed HPC data centers

Khan, Awais; Kim, Taeuk; Byun, Hyunki; Kim, Young-Jae

doi:10.1016/j.future.2019.06.006

Cited by 19 publications

(9 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…But, it poses several challenges at the same time. First, data sharing across applications and other collaborators is an essential requirement of the scientific community [27], [44]. It is challenging to access, select and share a PMO without additional descriptive metadata.…”

Section: E Motivationmentioning

confidence: 99%

“…MOSIQS enables scientists and applications to search and query PMOs using the metadata stored in the pool key-value store. However, despite the simpler API for metadata storage, an inherited limitation of key-value store is; lack of multiattribute or multi-dimensional search queries, which is a common trend in scientific applications and communities [2], [27], [30], [46]. Scientific data is largely unstructured and contains a lot of descriptive metadata in the form of keyvalue pair attributes [26], and retrieving the desired dataset usually depends on such multiple metadata attributes [30], [47].…”

Section: ) Object Sharing Controlsmentioning

confidence: 99%

“…However, pVM and DAOS-M lack scientific metadata indexing and querying for memory-level objects. Further, these studies also lack data sharing controls, an important requirement of scientific computing [27].…”

Section: Related Workmentioning

confidence: 99%

“…Scientific Data Indexing and Querying: Several studies have proposed various solutions and optimizations to improve the scientific indexing and querying tightly integrating into the file system or using external databases [2], [9], [25]- [27], [30]- [32], [46], [51]- [56]. SoMeta [51] provides scientific metadata querying via scalable self-described object and map objects to files internally at the file system level.…”

Section: Related Workmentioning

confidence: 99%

“…It is a common standard in the scientific community to tag or annotate data objects with additional descriptive metadata for a better understanding of data for collaborators [2], [25], [27]. Fourth, with ever-changing data analysis scenarios, to select a subset of PMOs from million and billions of PMOs in a shared PM pool based on metadata or user-defined tags without additional indexing becomes highly challenging [27]- [30]. These challenges drive the need for essentially effective scientific metadata search services and querying on top of persistent memory object storage abstraction.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

MOSIQS: Persistent Memory Object Storage With Metadata Indexing and Querying for Scientific Computing

et al. 2021

Self Cite

View full text Add to dashboard Cite

Scientific applications often require high-bandwidth shared storage to perform joint simulations and collaborative data analytics. Shared memory pools provide a chance to satisfy such needs. Recently, a high-speed network such as Gen-Z utilizing persistent memory (PM) offers an opportunity to create a shared memory pool connected to compute nodes. However, there are several challenges to use scientific applications on the shared memory pool directly such as scalability, failure-atomicity, and lack of scientific metadata-based search and query. In this paper, we propose MOSIQS, a persistent memory object storage framework with metadata indexing and querying for scientific computing. We design MOSIQS based on the key idea that memory objects on PM pool can live beyond the application lifetime and can become the sharing currency for applications and scientists. MOSIQS provides an aggregate memory pool atop an array of persistent memory devices to store and access memory objects to accelerate scientific computing. MOSIQS uses a lightweight persistent memory key-value store to manage the metadata of memory objects, which enables memory object sharing. To facilitate metadata search and query over millions of memory objects resident on memory pool, we introduce Group Split and Merge (GSM), a novel persistent index data structure designed primarily for scientific datasets. GSM splits and merges dynamically to minimize the query search space and maintains low query processing time while overcoming the index storage overhead. MOSIQS is implemented atop of PMDK. We evaluate the proposed approach on many-core server with an array of real PM devices. Experimental results show that MOSIQS gains a 100% write performance improvement and executes multi-attribute queries efficiently with 2.7× less index storage overhead offering significant potential to speed up scientific computing applications.

show abstract

Section: E Motivationmentioning

confidence: 99%