Distributed Resource Discovery in Large Scale Computing Systems

Gupta, Abhishek; Agrawal, Divyakant; Abbadi, Amr El

doi:10.1109/saint.2005.21

Cited by 19 publications

(9 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While SWORD provides scalability in querying, we have also addressed scalability in data collection and propagation. Several recent approaches have explored resource discovery in dynamic desktop Grid environments [8], [10], [22]. Most of these approaches use peer-to-peer query forwarding, and focus on finding single nodes for individual application tasks.…”

Section: Statistical Resource Guaranteesmentioning

confidence: 99%

Resource Bundles: Using Aggregation for Statistical Large-Scale Resource Discovery and Management

Cardosa

Chandra

2010

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

Resource discovery is an important process for finding suitable nodes that satisfy application requirements in large loosely-coupled distributed systems. Besides inter-node heterogeneity, many of these systems also show a high degree of intra-node dynamism, so that selecting nodes based only on their recently observed resource capacities can lead to poor deployment decisions resulting in application failures or migration overheads. However, most existing resource discovery mechanisms rely mainly on recent observations to achieve scalability in large systems. In this paper, we propose the notion of a resource bundle -a representative resource usage distribution for a group of nodes with similar resource usage patterns -that employs two complementary techniques to overcome the limitations of existing techniques: resource usage histograms to provide statistical guarantees for resource capacities, and clustering-based resource aggregation to achieve scalability. Using trace-driven simulations and data analysis of a month-long PlanetLab trace, we show that resource bundles are able to provide high accuracy for statistical resource discovery, while achieving high scalability. We also show that resource bundles are ideally suited for identifying group-level characteristics (e.g. hot spots, total group capacity).To automatically parameterize the bundling algorithm, we present an adaptive algorithm that can detect online fluctuations in resource heterogeneity.

show abstract

Section: Statistical Resource Guaranteesmentioning

confidence: 99%

Resource Bundles: Using Aggregation for Statistical Large-Scale Resource Discovery and Management

Cardosa

Chandra

2010

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

show abstract

“…For instance, Grid resource discovery systems based on both structured [51,52] and unstructured [53] have been proposed in the literature. As another example we mention Grido, a backbone overlay system for the provision of generic network services accessible via a Grid-based interface.…”

Section: Related Workmentioning

confidence: 99%

The File Mover: high‐performance data transfer for the grid

Anglano

Canonico

2007

Concurrency and Computation

View full text Add to dashboard Cite

SUMMARYThe exploration in many scientific disciplines (e.g. High-Energy Physics, Climate Modeling, and Life Sciences) involves the production and the analysis of massive data collections, whose archival, retrieval, and analysis require the coordinated usage of high-capacity computing, network, and storage resources. To obtain satisfactory performance, these applications require the availability of a high-performance, reliable data transfer mechanisms, able to minimize the data transfer time that often dominates their execution time. In this paper we present the File Mover, an efficient data transfer system specifically tailored to the needs of data-intensive applications, that exploits the overlay networks paradigm to provide superior performance with respect to conventional file transfer systems. An extensive experimental evaluation, carried out by means of a proof-of-concept implementation of the File Mover for a variety of network scenarios, shows the ability of the File Mover to outperform alternative data transfer systems. Copyright

show abstract

“…Given a generic multiprocessor topology and configuration, their goal is to come up with the optimal scheduling that minimizes the total processing time. On the other hand, researchers in the area of grid and P2P computing have focussed on identifying good peers [14], [11] based on the capability and reliability of their neighbors for possible resource sharing. As a result, the design of load sharing algorithms for large scale distributed systems is often isolated from the resource location process.…”

Section: Introductionmentioning

confidence: 99%

Computing Real Time Jobs in P2P Networks

Yao

Zhou

Bhuyan

2006

Proceedings. 2006 31st IEEE Conference on Local Computer Networks

View full text Add to dashboard Cite

Abstract-In this paper, we present a distributed computing framework designed to support higher quality of service and fault tolerance for processing deadline-driven tasks in a P2P environment. Our proposed strategy strives to build an open infrastructure that is accessible by ordinary users for both cycle donation and consumption. For jobs that fail to be locally accommodated, the proposed scheduler MET (Maximum Efficiency Tree) builds a dynamic multi-level resource tree with minimal yet sufficient power to process the job prior to its deadline. The peer selection policy is based on a joint evaluation of the computational power and communication bandwidth at the nodes. Further, with an optimal load sharing scheme, the resulting resource tree is guaranteed to be power efficient.The proposed computing protocol offers an approach for utilizing idle computing cycles of peer computers on the Internet in a P2P manner. The protocol exhibits three attractive features -decentralized operation, optimized load balancing and guaranteed resource utilization. Extensive simulation experiments are conducted to study the effectiveness of the proposed framework under various network conditions. We compare our strategy with two other tree construction algorithms, namely MST (Minimum Spanning Tree) and MCT (Maximum Computation Tree). It is demonstrated that MET outperforms both MST and MCT consistently. Further, sensitivity results with random node failure/join are also furnished.

show abstract

Distributed Resource Discovery in Large Scale Computing Systems

Cited by 19 publications

References 10 publications

Resource Bundles: Using Aggregation for Statistical Large-Scale Resource Discovery and Management

Resource Bundles: Using Aggregation for Statistical Large-Scale Resource Discovery and Management

The File Mover: high‐performance data transfer for the grid

Computing Real Time Jobs in P2P Networks

Contact Info

Product

Resources

About