Choosing a Data Storage Format in the Apache Hadoop System Based on Experimental Evaluation Using Apache Spark

Belov, V.; Tatarintsev, Andrey; Nikulchev, Evgeny

doi:10.3390/sym13020195

Cited by 13 publications

(3 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While the basic unit of information is very straightforward in a data file stored in characters (one byte equals one character), finding the actual data values is often much harder. This means that it is usually necessary to read the entire file to find any value [23][24][25].…”

Section: Binary Format Of Tsmld Storage Filesmentioning

confidence: 99%

Storage Method of Multi-Channel Lidar Data Based on Tree Structure

Chen

Gao

Zhu

et al. 2022

Preprint

View full text Add to dashboard Cite

The multi-channel lidar has the characteristics of fast acquisition speed, large data volume, high dimension, and strong real-time storage, which makes it difficult to be met using the traditional lidar data storage methods. This paper presents a novel approach to store and convert the multi-channel lidar data by traversal method of the tree structure and binary code. In the proposed approach, a tree structure is constructed based on the multi-dimensional characteristics of multi-channel lidar data and the hierarchical relationship between them. The adjacency table storage structure data in the memory is used to generate the sub-tree of the multi-channel lidar data. The results show that the proposed tree structure approach can save the storage capacity and improve the retrieval speed, which can meet the needs of efficient storage and retrieval of multi-channel lidar data.

show abstract

Section: Binary Format Of Tsmld Storage Filesmentioning

confidence: 99%

Storage Method of Multi-Channel Lidar Data Based on Tree Structure

Chen

Gao

Zhu

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…First of all, the massive amount of data has led to the reduction of information accuracy, and the change in the status and behavior of the relationship between things (Belov et al, 2021). The traditional accurate results are replaced by data that can be arranged (Nurnawati et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

Design and Application of Dance Teaching and Evaluation System Based on Big Data

Li,

Dong

2023

International Journal of Web-Based Learning and Teaching Technologies

View full text Add to dashboard Cite

In order to deeply analyze and evaluate the changes in the comprehensive quality of college students' sports dance, the overall idea of systematically evaluating the changes in the comprehensive quality of college students' sports dance was established. Firstly, this article uses the triangular fuzzy number method to measure the evaluation information of the evaluator's dance curriculum plan according to the evaluation indicators. Then, evaluation information from different groups and analytic hierarchy process (AHP) are used to determine the weight of each evaluation index. Finally, this article constructs a website system based on Python's Django framework. In the final system test and questionnaire survey, it was found that users were relatively satisfied with the accuracy of their evaluations and believed that the accurate performance of in-depth learning data analysis would help them save more time and effort in course selection.

show abstract

“…The Apache Spark framework is much faster because of its in-memory storage and distributed computation. Parquet and Avro file systems are used in Spark frameworks which are very fast and efficient for big data analytics [3,4]. The data is represented as a binary format in Parquet and Avro.…”

Section: Introductionmentioning

confidence: 99%

A Novel Big Data Storage Reduction Model for Drill Down Search

Ragavan¹,

Rubavathi²

2022

Computer Systems Science and Engineering

View full text Add to dashboard Cite

Multi-level searching is called Drill down search. Right now, no drill down search feature is available in the existing search engines like Google, Yahoo, Bing and Baidu. Drill down search is very much useful for the end user to find the exact search results among the huge paginated search results. Higher level of drill down search with category based search feature leads to get the most accurate search results but it increases the number and size of the file system. The purpose of this manuscript is to implement a big data storage reduction binary file system model for category based drill down search engine that offers fast multilevel filtering capability. The basic methodology of the proposed model stores the search engine data in the binary file system model. To verify the effectiveness of the proposed file system model, 5 million unique keyword data are stored into a binary file, thereby analysing the proposed file system with efficiency. Some experimental results are also provided based on real data that show our storage model speed and superiority. Experiments demonstrated that our file system expansion ratio is constant and it reduces the disk storage space up to 30% with conventional database/file system and it also increases the search performance for any levels of search. To discuss deeply, the paper starts with the short introduction of drill down search followed by the discussion of important technologies used to implement big data storage reduction system in detail.

show abstract

Choosing a Data Storage Format in the Apache Hadoop System Based on Experimental Evaluation Using Apache Spark

Cited by 13 publications

References 24 publications

Storage Method of Multi-Channel Lidar Data Based on Tree Structure

Storage Method of Multi-Channel Lidar Data Based on Tree Structure

Design and Application of Dance Teaching and Evaluation System Based on Big Data

A Novel Big Data Storage Reduction Model for Drill Down Search

Contact Info

Product

Resources

About