High-Performance Hybrid Key-Value Store on Modern Clusters with RDMA Interconnects and SSDs: Non-blocking Extensions, Designs, and Benefits

Shankar, Dipti; Lu, Xiaoyi; Islam, Nusrat Sharmin; Wasi-ur-Rahman, Md.; Panda, Dhabaleswar K.

doi:10.1109/ipdps.2016.112

Cited by 18 publications

(4 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In 2015, Islam et al [10] started by placing Memcached, an in-memory KV data store, between Lustre and HDFS to enable low latency and high throughput on reads. Their work continued with Shankar et al [14,15] who replaced internal Memcached operations to improve the performance. This approach differs from our proposal because their storage is not durable data might be lost on failures.…”

Section: Related Workmentioning

confidence: 99%

Evaluating the Benefits of Key-Value Databases for Scientific Applications

Santamaria

Oden

Gil

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The convergence of Big Data applications with High -Performance Computing requires new methodologies to store, manage and process large amounts of information. Traditional storage solutions are unable to scale and that results in complex coding strategies. For example, the brain atlas of the Human Brain Project has the challenge to process large amounts of high-resolution brain images. Given the computing needs, we study the effects of replacing a traditional storage system with a distributed Key-Value database on a cell segmentation application. The original code uses HDF5 files on GPFS through an intricate interface, imposing synchronizations. On the other hand, by using Apache Cassandra or ScyllaDB through Hecuba, the application code is greatly simplified. Thanks to the Key-Value data model, the number of synchronizations is reduced and the time dedicated to I/O scales when increasing the number of nodes.

show abstract

Section: Related Workmentioning

confidence: 99%

Evaluating the Benefits of Key-Value Databases for Scientific Applications

Santamaria

Oden

Gil

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…However, current-generation out-of-box big data analytics and management software stacks (e.g., Hadoop, Spark, Flink, Memcached) have not fully embraced such technologies. For instance, recent studies (Rahman et al, 2014, Islam et al, 2016b, Shankar, Lu, Islam, et al, 2016, Y. Wang et al, 2015, Lim et al, 2014, Huang et al, 2014, Arulraj et al, 2015 have shed light on the possible performance improvements for different big data middleware by taking advantage of RDMA over InfiniBand network, byte-addressability, and persistency of NVM. In this book, we will discuss more details about technological trends in modern HPC and data center clusters in chapter 4.…”

Section: Technological Trendsmentioning

confidence: 99%

“…These protocols are built with operating system-centric concepts and interfaces, such as Sockets, Portable Operating System Interface (POSIX), etc. These programming models typically have higher overhead due to the context switches and buffer copies between user-space and kernel-space (Rahman et al, 2014, Islam et al, 2016b, Shankar, Lu, Islam, et al, 2016. With more and more advanced technologies provided by the underlying hardware layer, new programming models and interfaces are becoming available and they can provide pure user-space and zero-copy communication and I/O protocols for the applications.…”

Section: Convergence In Hpc Big Data and Deep Learningmentioning

confidence: 99%

High-Performance Big Data Computing

Panda¹,

Lu²,

Shankar³

2022

View full text Add to dashboard Cite

An in-depth overview of an emerging field that brings together high-performance computing, big data processing, and deep learning. Over the last decade, the exponential explosion of data known as big data has changed the way we understand and harness the power of data. The emerging field of high-performance big data computing, which brings together high-performance computing (HPC), big data processing, and deep learning, aims to meet the challenges posed by large-scale data processing. This book offers an in-depth overview of high-performance big data computing and the associated technical issues, approaches, and solutions. The book covers basic concepts and necessary background knowledge, including data processing frameworks, storage systems, and hardware capabilities; offers a detailed discussion of technical issues in accelerating big data computing in terms of computation, communication, memory and storage, codesign, workload characterization and benchmarking, and system deployment and management; and surveys benchmarks and workloads for evaluating big data middleware systems. It presents a detailed discussion of big data computing systems and applications with high-performance networking, computing, and storage technologies, including state-of-the-art designs for data processing and storage systems. Finally, the book considers some advanced research topics in high-performance big data computing, including designing high-performance deep learning over big data (DLoBD) stacks and HPC cloud technologies.

show abstract

“…cells with arbitrary byte content) are stored on disk. This is a tradeoff between fast main memory and large disk storage (utilization of different storage types (Shankar et al 2016)). Cell data are written directly to the disk and their storage is subsequently marked as ‘dirty'.…”

Section: Architecture and Data Storage Modelsmentioning

confidence: 99%

RSDB: an easy to deploy open‐source web platform for remote sensing raster and point cloud data management, exploration and processing

et al. 2020

View full text Add to dashboard Cite

Monitoring biodiversity and its change over space and time is a key challenge for ecological research. Utilizing remote sensing data in ecological research has been shown to be a powerful complement to labor‐intensive field work to derive area‐wide biodiversity indices, which can also capture larger spatiotemporal areas. However, preprocessing different types of remote sensing data for specific studies often requires extensive computational skills as the algorithms for creating final data products are scattered across the literature. Our remote sensing database (RSDB, ) serves as an integrative storage, management and processing solution for several types of remote sensing data. Transparent data processing capabilities for more than 200 biodiversity‐related indices are included and can be used to calculate ready‐to‐use data products for a variety of ecological studies. These data products, for example, leaf area, vertical distribution of biomass and foliage height derived from light detection and ranging (LiDAR) point clouds, are individually calculated for user‐specified study areas and can serve as proxies for ecological variables, such as the availability of resources or habitat heterogeneity. The primary RSDB user interface is web browser based on management, exploration, visualization and product processing capabilities. The RSDB stores data in layers that are discoverable by textual metadata and mapping on overview maps. Individual data layers can be explored using interactive maps and 3‐D visualizations. We additionally provide a seamless connection of the RSDB's functionality with the R environment through our accompanying R package for scripting and advanced processing tasks. We designed the RSDB to be an easily accessible solution for on‐demand processing of comprehensive biodiversity indices. To date, the RSDB has been used and tested in several studies, and we are convinced that it will continue to facilitate biodiversity monitoring via its application to ecological research.

show abstract

High-Performance Hybrid Key-Value Store on Modern Clusters with RDMA Interconnects and SSDs: Non-blocking Extensions, Designs, and Benefits

Cited by 18 publications

References 12 publications

Evaluating the Benefits of Key-Value Databases for Scientific Applications

Evaluating the Benefits of Key-Value Databases for Scientific Applications

High-Performance Big Data Computing

RSDB: an easy to deploy open‐source web platform for remote sensing raster and point cloud data management, exploration and processing

Contact Info

Product

Resources

About