Adaptive Combiner for MapReduce on cloud computing

Huang, Tzu‐Chi; Chu, K.R.; Lee, Wei-Tsong; Ho, Yu-Sheng

doi:10.1007/s10586-014-0362-3

Cited by 7 publications

(3 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When evaluating collision risks for pairs of vessels, some pairs do not have to be examined because the likelihood of collision is very small. For selecting candidate pairs, we investigated indexing data structures like the R tree [21], PPR tree [22], and R * tree [23]. These data structures are efficient for entities that are stationary or nearly stationary, but the management cost is high in maritime traffic monitoring services in which vessels keep moving for a majority of their time.…”

Section: Adaptive Information Visualization Methods For Maritime Trmentioning

confidence: 99%

Adaptive Information Visualization for Maritime Traffic Stream Sensor Data with Parallel Context Acquisition and Machine Learning

Kim

Lee

2019

Sensors

View full text Add to dashboard Cite

Excessive information significantly increases the mental burden on operators of critical monitoring services such as maritime and air traffic control. In these fields, vessels and aircraft have sensors that transmit data to a control center. Because of the large volume of collected data, it is infeasible for monitoring stations to display all of the information on monitoring screens that have limited sizes. This paper proposes a method for automatically selecting maritime traffic stream data for display from a large number of candidates in a context-aware manner. Safety is the most important concern in maritime traffic control, and special care must be taken to avoid collisions between vessels at sea. It presents an architecture for an adaptive information visualization system for a maritime traffic control service. The proposed system adaptively determines the information to be displayed based on the safety evaluation scores and expertise of vessel traffic service operators. It also introduces a method for safety context acquisition to assess the risk of collisions between vessels, using parallel and distributed processing of maritime stream data transmitted by sensors on the vessels at sea. It provides an information-filtering, knowledge extraction method based on the work logs of traffic service operators, using a machine learning technique to generate a decision tree. We applied the proposed system architecture to a large dataset collected at a port. Our results indicate that the proposed system can adaptively select traffic information according to port conditions and to ensure safety and efficiency.

show abstract

Section: Adaptive Information Visualization Methods For Maritime Trmentioning

confidence: 99%

Adaptive Information Visualization for Maritime Traffic Stream Sensor Data with Parallel Context Acquisition and Machine Learning

Kim

Lee

2019

Sensors

View full text Add to dashboard Cite

show abstract

“…For instance, some applications require a certain type of sequential container. This is the case of MapReduce systems, where, by default, containers of the first phase (map) need to finish before the containers of the second phase (reduce) start [7,8].…”

Section: Constraintmentioning

confidence: 99%

Feedback-Based Resource Allocation in MapReduce-Based Systems

Memishi

Pérez

Antoniu

2016

Scientific Programming

View full text Add to dashboard Cite

Containers are considered an optimized fine-grain alternative to virtual machines in cloud-based systems. Some of the approaches which have adopted the use of containers are the MapReduce frameworks. This paper makes an analysis of the use of containers in MapReduce-based systems, concluding that the resource utilization of these systems in terms of containers is suboptimal. In order to solve this, the paper describes AdaptCont, a proposal for optimizing the containers allocation in MapReduce systems. AdaptCont is based on the foundations of feedback systems. Two different selection approaches, Dynamic AdaptCont and Pool AdaptCont, are defined. Whereas Dynamic AdaptCont calculates the exact amount of resources per each container, Pool AdaptCont chooses a predefined container from a pool of available configurations. AdaptCont is evaluated for a particular case, the application master container of Hadoop YARN. As we can see in the evaluation, AdaptCont behaves much better than the default resource allocation mechanism of Hadoop YARN.

show abstract

“…For instance, some applications require a certain type of sequential containers. This is the case of MapReduce systems, where by default, containers of the first phase (map) need to finish before the containers of the second phase (reduce) start [102,60].…”

Section: Constraint Filteringmentioning

confidence: 99%

Optimizing the reliability and resource efficiency of MapReduce-based systems

Memishi¹

View full text Add to dashboard Cite

Due to the increase of huge data volumes, a new parallel computing paradigm to process big data in an efficient way has arisen. Many of these systems, called dataintensive computing systems, follow the Google MapReduce programming model. The main advantage of these systems is based on the idea of sending the computation where the data resides, trying to provide scalability and efficiency.In failure-free scenarios, these frameworks usually achieve good results. However, these ones are not realistic scenarios. Consequently, these frameworks exhibit some fault tolerance and dependability techniques as built-in features. On the other hand, dependability improvements are known to imply additional resource costs. This is reasonable and providers offering these infrastructures are aware of this. Nevertheless, not all the approaches provide the same tradeoff between fault tolerant capabilities (or more generally, reliability capabilities) and cost.In this thesis, we have addressed the coexistence between reliability and resource efficiency in MapReduce-based systems, looking for methodologies that introduce the minimal cost and guarantee an appropriate level of reliability. In order to achieve this, we have proposed: (i) a formalization of a failure detector abstraction; (ii) an alternative solution to single points of failure of these frameworks, and finally (iii) a novel feedback-based resource allocation system at the container level.Finally, our generic contributions have been instantiated for the Hadoop YARN architecture, which is the state-of-the-art framework in the data-intensive computing systems community nowadays. The thesis demonstrates how all our approaches outperform Hadoop YARN in terms of reliability and resource efficiency. 3 ResumenDebido al gran incremento de datos digitales que ha tenido lugar en los últimos años, ha surgido un nuevo paradigma de computación paralela para el procesamiento eficiente de grandes volúmenes de datos. Muchos de los sistemas basados en este paradigma, también llamados sistemas de computación intensiva de datos, siguen el modelo de programación de Google MapReduce. La principal ventaja de los sistemas MapReduce es que se basan en la idea de enviar la computación donde residen los datos, tratando de proporcionar escalabilidad y eficiencia.En escenarios libres de fallo, estos sistemas generalmente logran buenos resultados. Sin embargo, la mayoría de escenarios donde se utilizan, se caracterizan por la existencia de fallos. Por tanto, estas plataformas suelen incorporar características de tolerancia a fallos y fiabilidad. Por otro lado, es reconocido que las mejoras en confiabilidad vienen asociadas a costes adicionales en recursos. Esto es razonable y los proveedores que ofrecen este tipo de infraestructuras son conscientes de ello. No obstante, no todos los enfoques proporcionan la misma solución de compromiso entre las capacidades de tolerancia a fallo (o de manera general, las capacidades de fiabilidad) y su coste.Esta tesis ha tratado la problemática de la coexist...

show abstract

Adaptive Combiner for MapReduce on cloud computing

Cited by 7 publications

References 15 publications

Adaptive Information Visualization for Maritime Traffic Stream Sensor Data with Parallel Context Acquisition and Machine Learning

Adaptive Information Visualization for Maritime Traffic Stream Sensor Data with Parallel Context Acquisition and Machine Learning

Feedback-Based Resource Allocation in MapReduce-Based Systems

Optimizing the reliability and resource efficiency of MapReduce-based systems

Contact Info

Product

Resources

About