Enabling lock-free concurrent fine-grain access to massive distributed data: Application to supernovae detection

Abstract

Abstract-We consider the problem of efficiently managing massive data in a large-scale distributed environment. We consider data strings of size in the order of Terabytes, shared and accessed by concurrent clients. On each individual access, a segment of a string, of the order of Megabytes, is read or modified. Our goal is to provide the clients with efficient finegrain access the data string as concurrently as possible, without locking the string itself. This issue is crucial in the context of applications in the field of astronomy, databases, data mining and multimedia. We illustrate these requiremens with the case of an application for searching supernovae. Our solution relies on distributed, RAM-based data storage, while leveraging a DHTbased, parallel metadata management scheme. The proposed architecture and algorithms have been validated through a software prototype and evaluated in a cluster environment.

BlobSeer

“…In its initial state, we assume any blob is considered empty (its size is 0) and is labeled with version 0. (Note that our previous work [13,12] was relying on different assumptions: the blob size was statically specified at the initialization time and could not be extended. )…”

“…Parallel and distributed file system also consider using objects for low-level storage [6,17,7]. In other scenarios, huge blobs need to be used concurrently at the highest level layers of applications directly: high-energy physics applications, multimedia processing [4] or astronomy [13].…”

“…Our approach has been implemented and experimented within our prototype, called BlobSeer: a binary large object management service. In previous work [13,12] we have handled versioning in a static way: blobs were considered huge storage objects of predefined, fixed sizes that are first allocated, then manipulated by reading and writing parts of them. However, in most real life scenarios, blobs need to dynamically grow, as new data is continuously gathered.…”

Enabling High Data Throughput in Desktop Grids through Decentralized Data and Metadata Management: The BlobSeer Approach

“…BlobSeer at a glance We have experimentally studied the approach outlined above using our BlobSeer prototype: a blob (Binary Large OBject) management service [13,14]. To cope with very large data blobs, BlobSeer uses striping: each blob is made up of blocks of a fixed size psize, referred to as pages.…”