On the Organization of Cluster Voting with Massive Distributed Streams

Abstract

Data processing is one of the important challenges on Big Data. In this paper we investigate optimal processing algorithm for massive data streams, propose a new processing algorithm called multi-buffer based majority algorithm. The algorithm maintains time complexity of O(n) and selects prevalent elements of frequencies as low as 1%. Our experiments indicate that multi-buffer based majority algorithm has improvements on both accuracy and efficiency. Moreover, we use multibuffer based algorithm to process data streams on single system and distributed system. These experiments indicate that using multi-buffer based algorithm can have better performance on distributed system. Moreover, we give explanations of the experiments' result and indicate several major factors which influence the result accuracy: stream size, element range in the stream, frequency of predominant elements and our buffer sets.

A Cyber-Physical Stream Algorithm for Intelligent Software Defined Storage

“…For random distribution determining a certain increment rate, the optimal decrement rate turns out to be presented by α = 0.99. In Figure. 6, we show the improvement of the CPS algorithm compared to previous work [6] Single-Buffer Based algorithm. The CPS algorithm performs better at providing very high probabilities of selecting the predominant item of frequencies as low as 2%.…”

“…Moreover, we introduce the algorithm of CPS and physical design that holds CPS. Section 3.2 covers the role of CPS and other majority algorithm presented in previous work [6]. In section 4, we present the results of our experiments on CPS with input of stream data in random uniform distribution and stream data follows Zipf's law distribution.…”

“…Moreover, we introduce a design of an analog physical device that holds the CPS algorithm that extracts prevalent items from streams of big data. In section 3.2 we show the major role of the CPS algorithm and Multi-Buffer Based algorithm [6] in the suggested structure of SoftwareDefined Storage [5] in more details.…”

“…In previous work [6], we introduced Multi-Buffer Based algorithm that has the ability of retrieving the top k most frequent items in big data stream in leaner time O(n) with very limited space and memory requirements. In this paper we introduced the CPS that also has the time complexity of O(n), and limited space and memory requirement to extract the most frequent item over a stream of big data.…”

“…The construction of the suggested Software-Defined Storage emulates the basic features of the suggested memory organization of the brain: multi-attribute cortical map, content-addressable access, and stream resolution of multiple responses. The envisioned Software-Defined Storage incorporates two developments: memory device for multi-attribute items that can be accessed by any combinations of attributes using FuzzyFind procedures [9] and massive distributed streaming for resolution of multiple responses [6].…”