Block Storage Optimization and Parallel Data Processing and Analysis of Product Big Data Based on the Hadoop Platform

Abstract

The traditional distributed database storage architecture has the problems of low efficiency and storage capacity in managing data resources of seafood products. We reviewed various storage and retrieval technologies for the big data resources. A block storage layout optimization method based on the Hadoop platform and a parallel data processing and analysis method based on the MapReduce model are proposed. A multireplica consistent hashing algorithm based on data correlation and spatial and temporal properties is used in the parallel data processing and analysis method. The data distribution strategy and block size adjustment are studied based on the Hadoop platform. A multidata source parallel join query algorithm and a multi-channel data fusion feature extraction algorithm based on data-optimized storage are designed for the big data resources of seafood products according to the MapReduce parallel frame work. Practical verification shows that the storage optimization and data-retrieval methods provide supports for constructing a big data resource-management platform for seafood products and realize efficient organization and management of the big data resources of seafood products. The execution time of multidata source parallel retrieval is only 32% of the time of the standard Hadoop scheme, and the execution time of the multichannel data fusion feature extraction algorithm is only 35% of the time of the standard Hadoop scheme.

Enhancement of NoSQL Database Performance Using Parallel Processing

“…Traditional computer paradigms are challenged by data growth and real-time processing. In particular, data storage and retrieval have grown exponentially (Wang, Cheng, Zhang, Leng, & Liu, 2021). Using several computer resources simultaneously has become common to overcome these difficulties.…”

Intelligent Identification over Power Big Data: Opportunities, Solutions, and Challenges

“…Reliability and persistency of data storage 1) Effective and valid data collection. The process of obtaining raw data from an external system or network is referred to as data collection [36,37]. Effective and valid data collection is necessary because inefficient and improper data collection will negatively impact subsequent processing.…”

Parallel flight test data processing method

“…The data volume of a single flight test reaches tens of GB. At the same time, with the gradual development of the civil aircraft test flight represented by the C919 aircraft, the total amount of civil aircraft test parameters has increased dramatically, and the total amount of iNET-X [2] data collected and recorded in single flight test network test has reached hundreds of GB [3]. The surge of flight test data brings great pressure to data processing.…”