A scheduling framework for large-scale, parallel, and topology-aware applications

Kravtsov, Valentin; Bar, Pavel; Carmeli, David; Schuster, Assaf; Swain, Martin

doi:10.1016/j.jpdc.2010.05.001

Cited by 7 publications

(3 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Authors in [7] have focused in the placement of worker processors and executors of a Storm topology in the available cluster nodes, with the main goal being the minimiza-tion of inter-node communication. Similar scheduling techniques were proposed in the works of [56], and [99]. These works are orthogonal to ours and can further enhance the working of Storm, increasing the overall system's performance.…”

Section: Distributed Stream Processingmentioning

confidence: 81%

Analysis of Spatiotemporal Data

2012

Spatial Data Analysis in Ecology and Agriculture Using R

View full text Add to dashboard Cite

In this thesis, we address major challenges in spatiotemporal data mining. Spatiotemporal data are data that relate to both space and time. Spatiotemporal data mining is the process of discovering patterns and extracting knowledge from spatiotemporal data. Examples of spatiotemporal data are trajectories of moving vehicles (i.e., GPS equipped buses or taxis) that move around the city reporting their GPS location along with the timestamp of each measurement, or animals' trajectories that are particularly important for zoologists in order to analyze the ecological behavior. The ubiquitous mobile devices generate massive datasets of GPS trajectories that are particularly difficult to interpret or analyse. Examples of spatiotemporal data mining include the extraction of moving patterns among a collection of moving objects. Another example of trajectory data mining, focusing in urban environments, is the traffic modelling. This refers to the analysis of the real-time traffic reports in order to extract the current traffic condition across the road network and the forecasting of the future traffic condition. Detecting and forecasting the traffic condition is particularly important in order to perform travel time estimation of a given path. Also, traditional centralized systems are unable to cope with the current massive-scale of spatiotemporal data, mainly because a single host is not able to receive and process all incoming data in real-time. Due to this reason, there is a shift in using Distributed Stream Processing Systems (DSPS).The contributions in this thesis focus on three main topic areas: (i) discovering corridors from GPS trajectories, (ii) hybrid travel time estimation and (iii) distributed complex event processing. Bellow we provide a short description of each of these topics: Discovering corridors from GPS trajectoriesWe aim at detecting paths that are frequently followed by the moving objects in the spatiotemporal database. Our main hypothesis is that even if the moving objects have different origins and destinations they are following common paths. The detections of such common paths is particularly demanding due to the complex nature of the spatiotemporal data. In this thesis we proposed a pipelined algorithm that discovers such frequent paths. The proposed technique first discovers areas that are frequently followed together by the moving objects and then discovers frequent paths following a greedy algorithm. Hybrid travel time estimationThis thesis also, focuses on the analysis of traffic condition across the road network and more specifically with the travel time estimation problem from one location of the road network to another given the path that will be followed and the time of departure. This problem is particularly challenging since the travel time depends on multiple factors that are difficult to be modeled. In this thesis we proposed a hybrid travel time estimation technique, named HTTE. Our technique performs travel time estimation for a given path (map-matched in the road network), using da...

show abstract

Section: Distributed Stream Processingmentioning

confidence: 81%

Analysis of Spatiotemporal Data

2012

Spatial Data Analysis in Ecology and Agriculture Using R

View full text Add to dashboard Cite

show abstract

“…Topology aware work stealing in HotSLAW [28] and load balancing in CHARM++ [17] minimise the cost of task migration but do not expose the topology to the programmer, and hence unlike HdpH cannot guarantee that tasks remain close to each other. While some GRID/cloud middleware like [15] exposes complex topologies, the architectures are very different from HPC and the schedulers typically aim to minimise the cost of inter-process communication rather than migration. Perhaps most closely related is a parallel Haskell [14] that exposes a two-level topology.…”

Section: Related Workmentioning

confidence: 99%

The HdpH DSLs for scalable reliable computation

Maier

Stewart

Trinder

2014

Proceedings of the 2014 ACM SIGPLAN Symposium on Haskell

View full text Add to dashboard Cite

The statelessness of functional computations facilitates both parallelism and fault recovery. Faults and non-uniform communication topologies are key challenges for emergent large scale parallel architectures. We report on HdpH and HdpH-RS, a pair of Haskell DSLs designed to address these challenges for irregular task-parallel computations on large distributed-memory architectures. Both DSLs share an API combining explicit task placement with sophisticated work stealing. HdpH focuses on scalability by making placement and stealing topology aware whereas HdpH-RS delivers reliability by means of fault tolerant work stealing.We present operational semantics for both DSLs and investigate conditions for semantic equivalence of HdpH and HdpH-RS programs, that is, conditions under which topology awareness can be transparently traded for fault tolerance. We detail how the DSL implementations realise topology awareness and fault tolerance. We report an initial evaluation of scalability and fault tolerance on a 256-core cluster and on up to 32K cores of an HPC platform.

show abstract

“…The computing workflow optimization problems in distributed environments under different constraints have been extensively studied in various disciplines [23,20,13,3,6,16,19,24,7] and continue to be the focus of distributed computing due to their theoretical significance and practical importance.…”

Section: Related Workmentioning

confidence: 99%

Optimizing end-to-end performance of data-intensive computing pipelines in heterogeneous network environments

2011

Journal of Parallel and Distributed Computing

View full text Add to dashboard Cite

A scheduling framework for large-scale, parallel, and topology-aware applications

Cited by 7 publications

References 13 publications

Analysis of Spatiotemporal Data

Analysis of Spatiotemporal Data

The HdpH DSLs for scalable reliable computation

Optimizing end-to-end performance of data-intensive computing pipelines in heterogeneous network environments

Contact Info

Product

Resources

About