Efficient updates for a shared nothing analytics platform

Abstract

In this paper we describe a cloud-based data-warehouselike system especially targeted to time series data. Apart from the benefits that a distributed storage built on top of a shared-nothing architecture offers, our system is designed to efficiently cope with continuous, on-line updates of temporally ordered data without compromising the query throughput. Through a totally customizable process performing asynchronous aggregation of past records, we achieve significant gains in storage and update times compared to traditional methods, maintaining a high accuracy in query responses for our target application. Experiments using our prototype implementation over an actual testbed prove that our scheme considerably accelerates (by a factor above 3) the update procedure and reduces required storage by at least 30%. We also show how these gains are related to the level and rate of aggregation performed.

What's the big deal about big data?

“…Position papers have suggested a need for substantial work in this area (e.g., [14,15,16]). Individual efforts have made piecemeal advances, for 40 NICK CERCONE, F'IEEE example by migrating analytic applications to the cloud for time series data [17] and by devising data warehouses for the cloud [18]. There is interest in using the MapReduce paradigm for analytics (e.g., [19,20]); for example, IBM Research reported efforts primarily focused on extracting and analyzing large-scale unstructured data to do search-driven analytics us ing the Hadoop platform [21].…”

Διαχείριση Δεδομένων Σε Κατανεμημένα Συστήματα Μεγάλης Κλίμακας Για Χρήση Σε Εφαρμογές Αναλυτικής Επεξεργασίας

“…Το Brown Dwarf [DTK10a,DTK10c,DTK10b] είναι ένα σύστημα το οποίο διανέμει μια κεντρική δομή δεικτοδότησης, το Dwarf [SDRK02] στους κόμβους ενός αδόμητου δικτύου "εν κινήσει" (on-the-y), επιταχύνοντας τόσο τη δημιουργία του όσο και την επίλυσης των ερωτημάτων χάρη στην παραλληλοποίηση που επιβάλλει. Τα ερωτήματα ανάλυσης καθώς και οι ενημερώσεις πραγματοποιούνται online μέσω των συνεργαζόμενων κόμβων του αδόμητου δικτύου επικάλυψης, εξαλείφοντας τη δαπανηρή συμβατική διαδικασία που συνήθως γίνεται ασύγχρονα.…”

“…Brown Dwarf [DTK10a,DTK10c,DTK10b] is a data analytics system that distributes multidimensional data over commodity network nodes, without the use of any proprietary tool.…”