Digital Filter Implementation in Hadoop Data Mining System

Czerwiński, Dariusz

doi:10.1007/978-3-319-19419-6_39

Cited by 7 publications

(3 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [46] we have shown that dynamic rule of merging speeds up the process of about 10%. In [47] we have proved that nonrecursive sorting makes the merge method flexible to various architectures, and in [48] our ideas were reported for Hadoop systems. A theoretical introduction to parallelization of sorting was presented by Cole [25].…”

Section: Related Workmentioning

confidence: 82%

Fully Flexible Parallel Merge Sort for Multicore Architectures

2018

View full text Add to dashboard Cite

The development in multicore architectures gives a new line of processors that can flexibly distribute tasks between their logical cores. These need flexible models of efficient algorithms, both fast and stable. A new line of efficient sorting algorithms can support these systems to efficiently use all available resources. Processes and calculations shall be flexibly distributed between cores to make the performance as high as possible. In this article we present a fully flexible sorting method designed for parallel processing. The idea we describe in this article is based on modified merge sort, which in parallel form is designed for multicore architectures. The novelty of this idea is in particular way of processing. We have developed a fully flexible method that can be implemented for a number of processors. The tasks are flexibly distributed between logical cores to increase the efficiency of sorting. The method preserves separation of concerns; therefore, each of the processors works separately without any cross actions and interruptions. The proposed method was described in theoretical way, examined in tests, and compared to other methods. The results confirm high efficiency and show that with each newly added processor sorting becomes faster and more efficient.

show abstract

Section: Related Workmentioning

confidence: 82%

Fully Flexible Parallel Merge Sort for Multicore Architectures

2018

View full text Add to dashboard Cite

show abstract

“…The study area is Nong'an ,the total area is 5400 square kilometers.In 2013, the county planting area of agricultural land reached 366,000 hectares,the coverage rate of fine varieties is above 98%,it is one of the important commodity grain bases in Jilin Province.Nong'an County annual average temperature of 4.7 degrees, frost free period of 145 days, rainfall of 507.7 mm, the effective accumulated temperature of 2800 degrees.Flat terrain, four distinct seasons, is a temperate continental climate.The soil in Nongan county is divided into 10 soil types and 20 sub types, 50 genera and 111 species of soil .The zonal soil is black soil, chernozem [7] .This data comes from the seven real-time data monitoring stations located in Nong'an County Kai'an Town,Hualong Town Chenjiadian Village,real-time monitoring network for intelligent agriculture,monitoring and collection of crop production environment information using wireless sensor networks,thus to realize the real-time processing and analysis of the big data of the intelligent agriculture.…”

Section: Data Sourcesmentioning

confidence: 99%

Research and Application of Spark Platform on Big Data Processing in Intelligent Agriculture of Jilin Province

Chen

Zhao

et al. 2019

Computer and Computing Technologies in Agriculture XI

View full text Add to dashboard Cite

Aiming at the demand of real-time massive data processing of Intelligent Agriculture in Jilin Province, this paper studies the big data processing of Intelligent Agriculture in Jilin Province based on Spark platform by acquiring real-time data through monitoring platform. This study first conducted the performance comparison experiment of Hadoop and Spark data processing platform,then used the Spark distributed cluster computing platform,real-time processing the big data of monitoring area. The experimental results show that the Spark platform speeds up 11.4 times faster than the Hadoop platform in the case of 100 million data sizes; and based on the Spark platform for real-time processing of big data intelligent agricultural monitoring network, not only provides memory calculations to reduce IO overhead, but also the results are faster and more accurate. The research results provide strong support for the implementation of precision agriculture technology in intelligent agriculture.

show abstract

“…We have proved that non-recursive version of sorting is more flexible for the implementations on various architectures. Results of our research were examined on Hadoop architectures [42].…”

Section: Related Workmentioning

confidence: 99%

Parallelization of Modified Merge Sort Algorithm

Marszałek

2017

Symmetry

View full text Add to dashboard Cite

Abstract:Modern architectures make possible development in new algorithms for large data sets and distributed computing. The newly proposed versions can benefit both from faster computing on the multi core architectures, and intelligent programming techniques that use efficient procedures available in the latest programming studios. Frequently used algorithms to sort arrays of data in NoSQL databases is merge sort, where as NoSQL we understand any database without typical SQL programming interpreter. The author describes how to use the parallelization of the sorting processes for the modified method of sorting by merging for large data sets. The subject of this research is the claim that the parallelization of the sorting method is faster and beneficial for multi-core systems. Presented results show how the number of processors influences the sorting performance. The results are presented in theoretical assumptions and confirmed in practical benchmark tests. The method is compared to other sorting methods like quick sort, heap sort, and merge sort to show potential efficiency.

show abstract

Digital Filter Implementation in Hadoop Data Mining System

Cited by 7 publications

References 16 publications

Fully Flexible Parallel Merge Sort for Multicore Architectures

Fully Flexible Parallel Merge Sort for Multicore Architectures

Research and Application of Spark Platform on Big Data Processing in Intelligent Agriculture of Jilin Province

Parallelization of Modified Merge Sort Algorithm

Contact Info

Product

Resources

About