Linux block IO

Bjørling, Matias; Axboe, Jens; Nellans, David; Bonnet, Philippe

doi:10.1145/2485732.2485740

Cited by 131 publications

(43 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The difference between each of these solutions is in the layers of abstractions (storage virtualization) and the networking interconnects (Fibre Channel, RCoE, RDMA, etc.) between the storage and host [8,10,11]. Figure 2 is the simplistic representation of the Linux I/O stack [10,12].…”

Section: Linux I/o Stackmentioning

confidence: 99%

“…between the storage and host [8,10,11]. Figure 2 is the simplistic representation of the Linux I/O stack [10,12].…”

Section: Linux I/o Stackmentioning

confidence: 99%

“…The virtual file system (VFS) provides abstractions for applications (processes) to access storage devices via system calls. The calls include a file descriptor and the location [10,13,14]. VFS locates and determines the storage device as well as the file system hosting the data, starting from a relative location.…”

Section: Linux I/o Stackmentioning

confidence: 99%

“…Operating at the block layer makes the solution independent of the file system layer above, enabling it with the flexibility to support multiple heterogeneous file systems simultaneously [8,10].…”

Section: Os Block Layer: Additional Featuresmentioning

confidence: 99%

“…The block layer has access to such attributes [8,10], thereby providing opportunities to exploit information for intelligent optimizations.…”

Section: Os Block Layer: Additional Featuresmentioning

confidence: 99%

See 4 more Smart Citations

Host managed contention avoidance storage solutions for Big Data

Mishra

Somani

2017

J Big Data

View full text Add to dashboard Cite

The performance gap between compute and storage is fairly considerable. This results in a mismatch between the application needs from storage and what storage can deliver. The full potential of storage devices cannot be harnessed till all layers of I/O hierarchy function efficiently. Despite advanced optimizations applied across various layers along the odyssey of data access, the I/O stack still remains volatile. The problems associated due to the inefficiencies in data management get amplified in Big Data shared resource environments. The Linux OS (host) block layer is the most critical part of the I/O hierarchy, as it orchestrates the I/O requests from different applications to the underlying storage. Unfortunately, despite it's significance, the block layer, essentially the block I/O scheduler, hasn't evolved to meet the needs of Big Data. We have designed and developed two contention avoidance storage solutions, collectively known as "BID: Bulk I/O Dispatch" in the Linux block layer specifically to suit multitenant, multi-tasking shared Big Data environments. Hard disk drives (HDDs) form the backbone of data center storage. The data access time in HDDs is majorly governed by disk arm movements, which usually occurs when data is not accessed sequentially. Big Data applications exhibit evident sequentiality but due to the contentions amongst other I/O submitting applications, the I/O accesses get multiplexed which leads to higher disk arm movements. BID schemes aim to exploit the inherent I/O sequentiality of Big Data applications to improve the overall I/O completion time by reducing the avoidable disk arm movements. In the first part, we propose a dynamically adaptable block I/O scheduling scheme BID-HDD for disk based storage. BID-HDD tries to recreate the sequentiality in I/O access in order to provide performance isolation to each I/O submitting process. Through trace driven simulation based experiments with cloud emulating MapReduce benchmarks, we show the effectiveness of BID-HDD which results in 28-52% lesser time for all I/O requests than the best performing Linux disk schedulers. In the second part, we propose a hybrid scheme BID-Hybrid to exploit SCM's (SSDs) superior random performance to further avoid contentions at disk based storage. BID-Hybrid is able to efficiently offload non-bulky interruptions from HDD request queue to SSD queue using BID-HDD for disk request processing and multi-q FIFO architecture for SSD. This results in performance gain of 6-23% for MapReduce workloads when compared to BID-HDD and 33-54% over best performing Linux scheduling scheme. BID schemes as a whole is aimed to avoid contentions for disk based storage I/Os following system constraints without compromising SLAs.

show abstract

Section: Linux I/o Stackmentioning

confidence: 99%

“…between the storage and host [8,10,11]. Figure 2 is the simplistic representation of the Linux I/O stack [10,12].…”

Section: Linux I/o Stackmentioning

confidence: 99%

Section: Linux I/o Stackmentioning

confidence: 99%

Section: Os Block Layer: Additional Featuresmentioning

confidence: 99%

“…The block layer has access to such attributes [8,10], thereby providing opportunities to exploit information for intelligent optimizations.…”

Section: Os Block Layer: Additional Featuresmentioning

confidence: 99%

See 3 more Smart Citations

Host managed contention avoidance storage solutions for Big Data

Mishra

Somani

2017

J Big Data

View full text Add to dashboard Cite

show abstract

MBSA: a lightweight and flexible storage architecture for virtual machines

Chen

et al. 2017

Concurrency and Computation

View full text Add to dashboard Cite

With the advantages of extremely high access speed, low energy consumption, nonvolatility, and byte addressability, nonvolatile memory (NVM) device has already been setting off a revolution in storage field. Conventional storage architecture needs to be optimized or even redesigned from scratch to fully explore the performance potential of NVM device. However, most previous NVM-related works only explore its low access latency and low energy consumption. Few works have been done to explore the appropriate way to use NVM device for improving virtual machine's storage performance. In this paper, we comprehensively evaluate and analyze conventional virtual machine's storage architecture. We find that, even with cutting-edge optimization technologies, virtual machine can only achieve 30% of NVM device's original performance. Based on this observation, we propose a memory bus-based storage architecture, which we named MBSA. Memory bus-based storage architecture can greatly shorten the length of virtual machine's storage input/output stack and improve NVM device's use flexibility. In addition, an efficient wear-leveling algorithm is proposed to prolong NVM device's lifespan. To evaluate the new architecture, we implement it as well as the wear-leveling algorithm on real hardware and software platform.Experimental results show that MBSA can provide a big performance improvement, about 2.55X, and the wear-leveling algorithm can efficiently balance write operations on NVM device with a negligible performance overhead (no more than 3%).

show abstract

LDM: Lineage-Aware Data Management in Multi-tier Storage Systems

Mishra

Somani

2019

Lecture Notes in Networks and Systems

View full text Add to dashboard Cite

We design and develop LDM, a novel data management solution to cater the needs of applications exhibiting the lineage property, i.e. in which the current writes are future reads. In such a class of applications, slow writes significantly hurt the over-all performance of jobs, i.e. current writes determine the fate of next reads. We believe that in a large scale shared production cluster, the issues associated due to data management can be mitigated at a way higher layer in the hierarchy of the I/O path, even before requests to data access are made. Contrary to the current solutions to data management which are mostly reactive and/or based on heuristics, LDM is both deterministic and pro-active. We develop block-graphs, which enable LDM to capture the complete time-based data-task dependency associations, therefore use it to perform life-cycle management through tiering of data blocks. LDM amalgamates the information from the entire data center ecosystem, right from the application code, to file system mappings, the compute and storage devices topology, etc. to make oracle-like deterministic data management decisions. With trace-driven experiments, LDM is able to achieve 29-52% reduction in over-all data center workload execution time. Moreover, by deploying LDM with extensive pre-processing creates efficient data consumption pipelines, which also reduces write and read delays significantly.Abstract. We design and develop LDM, a novel data management solution to cater the needs of applications exhibiting the lineage property, i.e. in which the current writes are future reads. In such a class of applications, slow writes significantly hurt the over-all performance of jobs, i.e. current writes determine the fate of next reads. We believe that in a large scale shared production cluster, the issues associated due to data management can be mitigated at a way higher layer in the hierarchy of the I/O path, even before requests to data access are made. Contrary to the current solutions to data management which are mostly reactive and/or based on heuristics, LDM is both deterministic and pro-active. We develop block-graphs, which enable LDM to capture the complete time-based data-task dependency associations, therefore use it to perform life-cycle management through tiering of data blocks. LDM amalgamates the information from the entire data center ecosystem, right from the application code, to file system mappings, the compute and storage devices topology, etc. to make oracle-like deterministic data management decisions. With trace-driven experiments, LDM is able to achieve 29% to 52% reduction in over-all data center workload execution time. Moreover, by deploying LDM with extensive pre-processing creates efficient data consumption pipelines, which also reduces write and read delays significantly.

show abstract

Linux block IO

Cited by 131 publications

References 21 publications

Host managed contention avoidance storage solutions for Big Data

Host managed contention avoidance storage solutions for Big Data

MBSA: a lightweight and flexible storage architecture for virtual machines

LDM: Lineage-Aware Data Management in Multi-tier Storage Systems

Contact Info

Product

Resources

About