Host managed contention avoidance storage solutions for Big Data

Abstract

The performance gap between compute and storage is fairly considerable. This results in a mismatch between the application needs from storage and what storage can deliver. The full potential of storage devices cannot be harnessed till all layers of I/O hierarchy function efficiently. Despite advanced optimizations applied across various layers along the odyssey of data access, the I/O stack still remains volatile. The problems associated due to the inefficiencies in data management get amplified in Big Data shared resource environments. The Linux OS (host) block layer is the most critical part of the I/O hierarchy, as it orchestrates the I/O requests from different applications to the underlying storage. Unfortunately, despite it's significance, the block layer, essentially the block I/O scheduler, hasn't evolved to meet the needs of Big Data. We have designed and developed two contention avoidance storage solutions, collectively known as "BID: Bulk I/O Dispatch" in the Linux block layer specifically to suit multitenant, multi-tasking shared Big Data environments. Hard disk drives (HDDs) form the backbone of data center storage. The data access time in HDDs is majorly governed by disk arm movements, which usually occurs when data is not accessed sequentially. Big Data applications exhibit evident sequentiality but due to the contentions amongst other I/O submitting applications, the I/O accesses get multiplexed which leads to higher disk arm movements. BID schemes aim to exploit the inherent I/O sequentiality of Big Data applications to improve the overall I/O completion time by reducing the avoidable disk arm movements. In the first part, we propose a dynamically adaptable block I/O scheduling scheme BID-HDD for disk based storage. BID-HDD tries to recreate the sequentiality in I/O access in order to provide performance isolation to each I/O submitting process. Through trace driven simulation based experiments with cloud emulating MapReduce benchmarks, we show the effectiveness of BID-HDD which results in 28-52% lesser time for all I/O requests than the best performing Linux disk schedulers. In the second part, we propose a hybrid scheme BID-Hybrid to exploit SCM's (SSDs) superior random performance to further avoid contentions at disk based storage. BID-Hybrid is able to efficiently offload non-bulky interruptions from HDD request queue to SSD queue using BID-HDD for disk request processing and multi-q FIFO architecture for SSD. This results in performance gain of 6-23% for MapReduce workloads when compared to BID-HDD and 33-54% over best performing Linux scheduling scheme. BID schemes as a whole is aimed to avoid contentions for disk based storage I/Os following system constraints without compromising SLAs.

LDM: Lineage-Aware Data Management in Multi-tier Storage Systems

“…Most studies have focused on studying data center operations to consolidate the computing needs and organize and optimize computing for multiple applications. Computing resources are believed to be abundant, but without appropriate attention, they are mostly waiting for data and wasting cycles [3] [8] [9] [29] [14] [30]. Moreover, for lineage based applications, the impact is more severe due to data-dependency between tasks.…”

“…Data centers today cater to a wide diaspora of applications which process multiple data sets for multiple jobs in a multi-user environment concurrently. They also deploy storage systems organized in multiple heterogeneous tiers, which is necessary to achieve cost-performance-capacity trade-off [3] [4] [5]. Dedicating physical resources for every application is not economically feasible.…”

“…Dedicating physical resources for every application is not economically feasible. Resource sharing causes contention affecting the efficiency and performance [3] [6] [7]. Despite advanced optimizations applied across the various layers along the odyssey of data access, the data management layer remains volatile [8] [9].…”

“…Storage media across all nodes with similar I/O characteristics form a tier 3. Tiering refers to orchestrating data between heterogeneous tiers of storage by leveraging individual strengths of each to maintain balance between Cost, Performance and Capacity.…”

Teaching Intelligence System Based on the Cloud Platform of the Internet of Things and Its Application in Physical Education

“…A genetic algorithm is used to cross mutate these code fragments to generate candidate titles, and test sets are used to detect the correctness of the output titles [8]. Mishra and Somani proposed the PAR method, which assumes that the materials for repairing errors exist in the source code of other projects and generates candidate titles by manually defining repair templates and randomly selecting constrained mutation operation, which overcomes the shortage of generating meaningless candidate titles by random mutation based on genetic algorithm [9,10]. Belyaev and Ray think that the ability of different repair templates is not the same, so high-frequency repair templates are preferred to generate candidate titles, which improve the accuracy of title generation compared with the PAR method [11,12].…”