Big Data Processing on Volunteer Computing

Abstract

In order to calculate the node big data contained in complex networks and realize the efficient calculation of complex networks, based on voluntary computing, taking ICE middleware as the communication medium, the loose coupling distributed framework DCBV based on voluntary computing is proposed. Then, the Master, Worker, and MiddleWare layers in the framework, and the development structure of a DCBV framework are designed. The task allocation and recovery strategy, message passing and communication mode, and fault tolerance processing are discussed. Finally, to calculate and verify parameters such as the average shortest path of the framework and shorten calculation time, an improved accurate shortest path algorithm, the N-SPFA algorithm, is proposed. Under different datasets, the node calculation and performance of the N-SPFA algorithm are explored. The algorithm is compared with four approximate shortest-path algorithms: Combined Link and Attribute (CLA), Lexicographic Breadth First Search (LBFS), Approximate algorithm of shortest path length based on center distance of area division (CDZ), and Hub Vertex of area and Core Expressway (HEA-CE). The results show that when the number of CPU threads is 4, the computation time of the DCBV framework is the shortest (514.63 ms). As the number of CPU cores increases, the overall computation time of the framework decreases gradually. For every 2 additional CPU cores, the number of tasks increases by 1. When the number of Worker nodes is 8 and the number of nodes is 1, the computation time of the framework is the shortest (210,979 ms), and the IO statistics data increase with the increase of Worker nodes. When the datasets are Undirected01 and Undirected02, the computation time of the N-SPFA algorithm is the shortest, which is 4520 ms and 7324 ms, respectively. However, the calculation time in the ca-condmat_undirected dataset is 175,292 ms, and the performance is slightly worse. Overall, however, the performance of the N-SPFA and SPFA algorithms is good. Therefore, the two algorithms are combined. For networks with less complexity, the computational scale coefficient of the SPFA algorithm can be set to 0.06, and for general networks, 0.2. When compared with other algorithms in different datasets, the pretreatment time, average query time, and overall query time of N-SPFA algorithm are the shortest, being 49.67 ms, 5.12 ms, and 94,720 ms, respectively. The accuracy (1.0087) and error rate (0.024) are also the best. In conclusion, voluntary computing can be applied to the processing of big data, which has a good reference significance for the distributed analysis of large-scale complex networks.

Parallel Processing Distributed-Memory Approach Influences on Performance of Multicomputer-Multicore Systems Using Single-Process Single-Thread

“…In order to implement a more sophisticated justice policy, it was also feasible to change the unit that distributes resources in the NIC's output during the scheduling algorithm's response phase. integrated link and attribute, lexicographic breadth first search, approximation algorithm for shortest route length based on center-division zone distance, and axis-vertex for area and principal highway are some of the methods used by Z. Lv, D. Chen, and A. K. Singh [27] in 2021. The findings demonstrate that the calculation time for the framework is the fastest with 4 CPU threads (514.63 ms).…”

Edge Intelligence-Assisted Asymmetrical Network Control and Video Decoding in the Industrial IoT with Speculative Parallelization

“…In the possible parallel dividing, apart from parallel compilation techniques, a sufficiently large memory is required as hardware. In particular, the data to be divided must reside in the memory and be uniformly executed via the bus system [26]. More precisely, the main thread is in the pending state and instructs the parallel task by monitoring the memory state.…”