Comparative Characteristics of Big Data Storage Formats

Belov, V.; Tatarintsev, Andrey; Nikulchev, Evgeny

doi:10.1088/1742-6596/1727/1/012005

Cited by 5 publications

(3 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Delta Lake is a storage layer for improving the reliability of data lakes [36][37][38]. Delta Lake can operate on the basis of implemented data lakes using Apache Hadoop [11], Amazon S3 [32] or Azure Data Lake Storage [39].…”

Section: Delta Lakementioning

confidence: 99%

See 1 more Smart Citation

Analysis of Big Data Storage Tools for Data Lakes based on Apache Hadoop Platform

Belov¹,

Nikulchev²

2021

IJACSA

Self Cite

View full text Add to dashboard Cite

When developing large data processing systems, the question of data storage arises. One of the modern tools for solving this problem is the so-called data lakes. Many implementations of data lakes use Apache Hadoop as a basic platform. Hadoop does not have a default data storage format, which leads to the task of choosing a data format when designing a data processing system. To solve this problem, it is necessary to proceed from the results of the assessment according to several criteria. In turn, experimental evaluation does not always give a complete understanding of the possibilities for working with a particular data storage format. In this case, it is necessary to study the features of the format, its internal structure, recommendations for use, etc. The article describes the features of both widely used data storage formats and the currently gaining popularity.

show abstract

Section: Delta Lakementioning

confidence: 99%

“…This file system is cheaper for use than commercial data bases. Using such data warehouse, choosing the right file format is critical [11]. File format determines how information would be stored in HDFS.…”

Section: Introductionmentioning

confidence: 99%

Analysis of Big Data Storage Tools for Data Lakes based on Apache Hadoop Platform

Belov¹,

Nikulchev²

2021

IJACSA

Self Cite

View full text Add to dashboard Cite

show abstract

“…Current storage media is struggling to keep pace with this growth rate and lacks sufficient storage capacity, presenting significant challenges to current data centers and storage techniques. To address this issue, there has been a surge in research and development of Big Data platforms and systems, resulting in the emergence of various software products, tools, and database systems to meet the escalating demand for large-scale data storage and processing [1]- [4].…”

Section: Introductionmentioning

confidence: 99%

DNA technology for big data storage and error detection solutions: Hamming code vs Cyclic Redundancy Check (CRC)

Sais,

Rafalia,

Abouchabaka

2023

E3S Web Conf.

View full text Add to dashboard Cite

There is an increasing need for high-capacity, highdensity storage media that can retain data for a long time, due to the exponential development in the capacity of information generated. The durability and high information density of synthetic deoxyribonucleic acid (DNA) make it an attractive and promising medium for data storage. DNA data storage technology is expected to revolutionize data storage in the coming years, replacing various Big Data storage technologies. As a medium that addresses the need for high-latency, immutable information storage, DNA has several potential advantages. One of the key advantages of DNA storage is its extraordinary density. Theoretically, a gram of DNA can encode 455 exabytes, or 2 bits per nucleotide. Unlike other digital storage media, synthetic DNA enables large quantities of data to be stored in a biological medium. This reduces the need for traditional storage media such as hard disks, which consume energy and require materials such as plastic or metals, and also often leads to the generation of electronic waste when they become obsolete or damaged. Additionally, although DNA degrades over thousands of years under non-ideal conditions, it is generally readable. Furthermore, as DNA possesses natural reading and writing enzymes as part of its biological functions, it is expected to remain the standard for data retrieval in the foreseeable future. However, the high error rate poses a significant challenge for DNA-based information coding strategies. Currently, it is impossible to execute DNA strand synthesis, amplification, or sequencing errors-free. In order to utilize synthetic DNA as a storage medium for digital data, specialized systems and solutions for direct error detection and correction must be implemented. The goal of this paper is to introduce DNA storage technology, outline the benefits and added value of this approach, and present an experiment comparing the effectiveness of two error detection and correction codes (Hamming and CRC) used in the DNA data storage strategy.

show abstract

Improved storage of Big Data in DNA using the PCA algorithm

En-Nattouh,

Jourani

2024

2024 International Conference on Circuit, Systems and Communication (ICCSC)

View full text Add to dashboard Cite

Comparative Characteristics of Big Data Storage Formats

Cited by 5 publications

References 6 publications

Analysis of Big Data Storage Tools for Data Lakes based on Apache Hadoop Platform

Analysis of Big Data Storage Tools for Data Lakes based on Apache Hadoop Platform

DNA technology for big data storage and error detection solutions: Hamming code vs Cyclic Redundancy Check (CRC)

Improved storage of Big Data in DNA using the PCA algorithm

Contact Info

Product

Resources

About