A Tickless AMP Distributed Core-Based Microkernel for Big Data

Sobh, Karim; El-Kadi, Amr

doi:10.1007/978-3-030-32520-6_41

Cited by 1 publication

(3 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, for completion purposes, we are presenting several experiments to assess the performance of a primitive prototype implementation that were conducted and presented earlier in [2]. The promising results were the main drive behind the most recent extensions of the micro-kernel design.…”

Section: Methodsmentioning

confidence: 99%

“…When the mappers finish their tasks, they notify the application master using IPIoE messages to start up the reducers and hence the shuffling phase. Data partitioning and network transfer over FDNTP [2], Fast Data Transfer Network Protocol, is started to assign each reducer a subset of the data. Each reducer launches its quick sort engine to sort its subset of the data and store its output on disk.…”

Section: Methodsmentioning

confidence: 99%

“…Given the exponential growth of data, big data processing is increasingly becoming one of the most challenging problems in the computing domain [1]. General-purpose operating systems are not designed to cater for big data fundamental processing, data distribution and communication needs, namely scalability and performance [2]. They cannot accurately identify the workload profile of running processes through heuristics.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

NileOS: A Distributed Asymmetric Core-Based Micro-Kernel for Big Data Processing

et al. 2021

Self Cite

View full text Add to dashboard Cite

Big Data applications have demanding expectations on computational resources front. Thus, general purpose operating systems are not a good fit. In this paper, we present a new special purpose distributed micro-kernel designed with big data applications' needs in mind. The new micro-kernel adopts a core-based Asymmetric Multiprocessing (AMP) approach. It optimizes interrupt management and I/O to suit the Map-Reduce model. The proposed micro-kernel design is based on Inter-processor Interrupts over Ethernet (IPIoE) frames and a BareMetal Operating System Markup Language (BOSML). A transparent deployment mechanism is presented to completely shield the developer of the micro-kernel service from the underlying distribution infrastructure and decouple the application implementation from its deployment perspective. Based on the initial prototype and the experiments presented, a considerable gain in performance of average 2.34 folds was achieved using the distributed TeraSort benchmark over Linux/Hadoop.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation