Data Processing on Distributed Systems Storage Challenges

Eddoujaji, Mohamed; Samadi, Hassan; Bohorma, Mohamed

doi:10.1007/978-981-16-3637-0_56

Cited by 5 publications

(3 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is an inevitable requirement for the development of technology and intelligence. Data mining can comprehensively use relevant algorithms to process a large amount of data and discover its hidden valuable information, which can realize the rapid conversion of data to knowledge and then value [ 28 , 29 , 30 , 31 , 32 ]. However, the current data volume is growing rapidly, and traditional algorithms based on single-node serial mining are no longer suitable for massive data requirements.…”

Section: Introductionmentioning

confidence: 99%

“…At present, the application research of cloud computing-based big data technology in the power industry is still in the exploratory stage [ 30 ]. Presently, most cloud computing platforms designed for monitoring centers are based on a single Hadoop framework and have certain limitations storing data before centralized processing that lead to processing time delay is too long [ 29 ], which cannot meet the requirements of online monitoring data [ 28 , 42 ]. Meet the data flow quickly and efficiently will become a trend of information processing in the future [ 12 , 68 ].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Big Data Analytics Using Cloud Computing Based Frameworks for Power Management Systems: Status, Constraints, and Future Recommendations

AL-Jumaili

Muniyandi

Hasan

et al. 2023

Sensors

View full text Add to dashboard Cite

Traditional parallel computing for power management systems has prime challenges such as execution time, computational complexity, and efficiency like process time and delays in power system condition monitoring, particularly consumer power consumption, weather data, and power generation for detecting and predicting data mining in the centralized parallel processing and diagnosis. Due to these constraints, data management has become a critical research consideration and bottleneck. To cope with these constraints, cloud computing-based methodologies have been introduced for managing data efficiently in power management systems. This paper reviews the concept of cloud computing architecture that can meet the multi-level real-time requirements to improve monitoring and performance which is designed for different application scenarios for power system monitoring. Then, cloud computing solutions are discussed under the background of big data, and emerging parallel programming models such as Hadoop, Spark, and Storm are briefly described to analyze the advancement, constraints, and innovations. The key performance metrics of cloud computing applications such as core data sampling, modeling, and analyzing the competitiveness of big data was modeled by applying related hypotheses. Finally, it introduces a new design concept with cloud computing and eventually some recommendations focusing on cloud computing infrastructure, and methods for managing real-time big data in the power management system that solve the data mining challenges.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Big Data Analytics Using Cloud Computing Based Frameworks for Power Management Systems: Status, Constraints, and Future Recommendations

AL-Jumaili

Muniyandi

Hasan

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

“…Currently, most organizations start the journey of transforming to DIS, but very few succeed in achieving optimal results [2,3]. Since the current DIS implementation effort is focused on the "need for speed" paradigm, including faster deployment, faster release cycles, and faster recovery, in fact, DIS development from scratch is a significant effort for organizations [5]. The recent research consistently reveals a reasonable research gap for a holistic DIS development platform.…”

Section: Introductionmentioning

confidence: 99%

Reference Architecture for Blockchain-Native Distributed Information System

et al. 2023

View full text Add to dashboard Cite

The blockchain paradigm and its application of Distributed Ledger Technology (DLT) are now enabling technologies for the worldwide development of trustworthy and decentralized solutions. DLT offers enormous potential to accommodate the intricate architectural patterns of individual Distributed Information Systems (DIS). Despite its immense potential, the current research effort is not focused on deploying DLT inside the DIS area. We identified sixteen essential DIS Architectural Properties (APs) that offer issues for DIS system architects using Grounded Theory. Then, we assessed the capabilities of the major DLT platform and highlighted its prospects and limits with regard to supporting DIS APs. Using the meta-analysis research technique, we demonstrated that DLT might serve as a development platform for a blockchain-native DIS (BN-DIS) that enables design and development processes. A BN-DIS utilizes DLT infrastructure and blockchain design principles to expand internationally, support thousands of distributed nodes, and employ DIS APs unique to the Blockchain. Lastly, we give a case study illustrating how DLT handles DIS APs in a real-world context, giving empirical proof of the DLT's efficacy in servicing DIS APs.

show abstract

An archive‐based method for efficiently handling small file problems in HDFS

Liu,

Jin,

Wang

et al. 2024

Concurrency and Computation

View full text Add to dashboard Cite

SummaryHadoop distributed file system (HDFS) performs well when storing and managing large files. However, its performance significantly decreases when dealing with massive small files. In response to this problem, a novel archive‐based solution is proposed. The archive refers to merging multiple small files into larger data files, which can effectively reduce the memory usage of the NameNode. The current archive‐based solutions have the disadvantages of long access time, long archive construction time, and no support for storage, updating and deleting small files in the archive system. Our method utilizes a dynamic hash function to distribute the metadata of small files across multiple metadata files. We construct a primary index that combines dynamic and static indexes for these metadata files. Regarding data files, include some read‐only files and one readable–writable file. A small file's contents are written into a readable and writable file. Upon reaching a predetermined threshold, the readable–writable file transitions into read‐only status, with a fresh readable–writable file replacing it. Experimental results show that the scheme improves the efficiency of archive access and archive creation and is more efficient than the original HDFS storage and update efficiency.

show abstract

Data Processing on Distributed Systems Storage Challenges

Cited by 5 publications

References 13 publications

Big Data Analytics Using Cloud Computing Based Frameworks for Power Management Systems: Status, Constraints, and Future Recommendations

Big Data Analytics Using Cloud Computing Based Frameworks for Power Management Systems: Status, Constraints, and Future Recommendations

Reference Architecture for Blockchain-Native Distributed Information System

An archive‐based method for efficiently handling small file problems in HDFS

Contact Info

Product

Resources

About