Scaling of Complex Calculations over Big Data-Sets

Grzegorowski, Marek

doi:10.1007/978-3-319-09912-5_7

Cited by 9 publications

(9 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The feature extraction was based on the sliding window [24] method and was configured to accept on its input a data set containing readings from multiple streams [5]. According to the submitted configuration each stream, stored in a row of the csv file (training and test set), was divided into three non-overlapping frames.…”

Section: Feature Extractionmentioning

confidence: 99%

“…The inter-window stats are presented in Table III. In order to optimize time-effectiveness of the experimentation process the framework implementation was modified so that it could be run in several modes, including: incremental stream processing [5], single-threaded mode and the map-reduce mode. In order to allow multi-mode framework operation, the feature extraction mechanism was isolated as a separate tool with respect to the runtime environment.…”

Section: Feature Extractionmentioning

confidence: 99%

“…Therefore, in order to simplify and speed up the feature extraction we introduced a novel framework designed to process streams of numerical readings from multiple sensors. The developed framework is ready to operate in production environments and, hence, is tailored for incremental processing of the emerging data based on the sliding window technique and the concept of parallelization presented in [5].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Massively Parallel Feature Extraction Framework Application in Predicting Dangerous Seismic Events

Grzegorowski

2016

Annals of Computer Science and Information Systems

Self Cite

View full text Add to dashboard Cite

Abstract-In this paper we introduce an automated mechanism for knowledge discovery from data streams. As a part of this work, we also present a new approach to the creation of classifiers ensemble based on a wide variety of models. Furthermore, we describe an innovative, highly scalable feature extraction and selection framework designed to work with the MapReduce programming model and the application of designed framework to build an ensemble of classifiers which takes into account both the quality and the diversity of individual models. The effectiveness of the solution has been verified through a participation in an open data mining competition which concerned the problem of predicting periods of increased seismic activity causing lifethreatening accidents in coal mines. The submitted solution obtained the highest AUC score of all the solutions uploaded by 106 participating research teams.

show abstract

Section: Feature Extractionmentioning

confidence: 99%

Section: Feature Extractionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Massively Parallel Feature Extraction Framework Application in Predicting Dangerous Seismic Events

Grzegorowski

2016

Annals of Computer Science and Information Systems

Self Cite

View full text Add to dashboard Cite

show abstract

Section: Related Workmentioning

confidence: 99%

“…The issue of mining industry devices diagnostics was raised among others in the works [7,10,18,25,31]. Besides, some initial concepts of the system that processes data streams delivered by the monitoring systems were presented in [6].However, to the best of the authors knowledge there is no example of the integrated decision support system for monitoring processes, devices and hazards in a coal mine (except the work dealing with DSS for coal transportation [14], which loosely corresponds to the given topic). …”

mentioning

confidence: 99%

Decision support and maintenance system for natural hazards, processes and equipment monitoring

Kozielski

Sikora

Wróbel

2016

EiN

View full text Add to dashboard Cite

Article citation info: IntroductionCoal mining is a heavy industry that plays an important role on an energy market and employs hundreds of thousands of people. Coal mining is also an industry, where large amount of data is produced but little is done to utilise them in further analysis. Besides, there is a justified need to integrate different aspects of coal mine operation in order to maintain continuity of mining what can be done by introduction of a decision support system (DSS).Currently coal mines are well equipped with the monitoring, supervising and dispatching systems connected with machines, devices and transport facilities. Additionally, there are the systems for monitoring natural hazards (methane-, seismic-and fire hazards) operating in the coal mines. All these systems are provided by many different companies, what causes problems with quality, integration and proper interpretation of the collected data. Another issue is that the collected data are used chiefly for current (temporary) visualisation on boards which display certain places in the mine. Whereas, application of domain knowledge and the results of historical data analysis can improve the operator's and supervisor's work significantly.For example, due to the short-term prognoses about methane concentration, linked with the information about the location and work intensity of the cutter loader, it is possible to prevent emergency energy shutdowns and maintain continuity of mining (the research on this methodology was discussed in [27]). This will enable to increase the production volume and to reduce the wear of electrical elements whose exploitation time depends on the number of switch-ons and switch-offs.It is possible to see the rising awareness of monitoring systems suppliers who has started to understand the necessity to make the next step in these systems development. Therefore, the companies providing monitoring systems seek their competitive advantage in equipping their systems with knowledge engineering, modelling and data analysis methods. This is a strong motivation to consider a DSS presented in this paper.The goal of this paper is to present an architecture of the DISESOR integrated decision support system. The system integrates data from different monitoring systems and contains an expert system module, that can utilise domain expert knowledge, and analytical module, that can be applied to diagnosis of the processes and devices and to prediction of natural hazards. Special focus of the paper is put on the data integration and data cleaning issues, such as outlier detection, realised by means of the data warehouse and the ETL process. The work also contains a more detailed presentation of the prediction module and two case studies showing real applications of the system.The contribution of the paper consists of the architecture of the DISESOR integrated decision support system, its data repository and prediction module. Additionally, it covers the presentation of the issues connected with the preparation and cleaning of the da...

show abstract

Window-Based Feature Engineering for Prediction of Methane Threats in Coal Mines

Grzegorowski

Stawicki

2015

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Scaling of Complex Calculations over Big Data-Sets

Cited by 9 publications

References 9 publications

Massively Parallel Feature Extraction Framework Application in Predicting Dangerous Seismic Events

Massively Parallel Feature Extraction Framework Application in Predicting Dangerous Seismic Events

Decision support and maintenance system for natural hazards, processes and equipment monitoring

Window-Based Feature Engineering for Prediction of Methane Threats in Coal Mines

Contact Info

Product

Resources

About