Treating Missing Data in Industrial Data Analytics

Ehrlinger, Lisa; Grubinger, Thomas; Varga, Bence; Pichler, Mario; Natschläger, Thomas; Zeindl, Jürgen

doi:10.1109/icdim.2018.8846984

Cited by 23 publications

(11 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Line pattern (LP) is caused by a breakdown of multiple sensors and after suitable rearrangements of columns seen as a "line" of MD in M (cf. [7,13]). Based on our experience from steel industry, we further distinguish between three LP subtypes: perfect, noisy, and asymmetric line pattern (see [6] for details).…”

Section: Patterns -Related Workmentioning

confidence: 99%

“…Reasons for MD are manifold: data integration from multiple sources, where not each attribute is populated by every source, incompletely gathered data in surveys, or technical issues like sensor breakdowns in industrial applications [4,7]. Since data disappears seldomly in a random fashion, MD can introduce serious bias into data analytics and it is thus crucial to detect and handle MD patterns appropriately.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Missing Data Patterns: From Theory to an Application in the Steel Industry

Bechný

Sobieczky

Zeindl

et al. 2021

33rd International Conference on Scientific and Statistical Database Management

Self Cite

View full text Add to dashboard Cite

Missing data (MD) is a prevalent problem and can negatively affect the trustworthiness of data analysis. In industrial use cases, faulty sensors or errors during data integration are common causes for systematically missing values. The majority of MD research deals with imputation, i.e., the replacement of missing values with "best guesses". Most imputation methods require missing values to occur independently, which is rarely the case in industry. Thus, it is necessary to identify missing data patterns (i.e., systematically missing values) prior to imputation (1) to understand the cause of the missingness, (2) to gain deeper insight into the data, and (3) to choose the proper imputation technique. However, in literature, there is a wide varity of MD patterns without a common formalization. In this paper, we introduce the first formal definition of MD patterns. Building on this theory, we developed a systematic approach on how to automatically detect MD patterns in industrial data. The approach has been developed in cooperation with voestalpine Stahl GmbH, where we applied it to real-world data from the steel industry and demonstrated its efficacy with a simulation study. CCS CONCEPTS• Information systems → Data cleaning; • Mathematics of computing → Multivariate statistics.

show abstract

Section: Patterns -Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Missing Data Patterns: From Theory to an Application in the Steel Industry

Bechný

Sobieczky

Zeindl

et al. 2021

33rd International Conference on Scientific and Statistical Database Management

Self Cite

View full text Add to dashboard Cite

show abstract

“…The power demand has 744 samples, and it has no missing samples. Since the missing data rate, a simple process of filling the missing data was performed using the simplest principle, linear interpolation [5]. Due to the granularity of the data, time-series with different sample frequency, and to avoid adding noise or losing information, the time series were not resampled.…”

Section: B Pre-processing Datamentioning

confidence: 99%

Descriptive Statistical Analysis of Frequency control-related variables of Nordic Power System

Acosta

Andrade

Vázquez

et al. 2020

2020 IEEE Power &Amp; Energy Society General Meeting (PESGM)

View full text Add to dashboard Cite

“…LCS, however, are prone to various failures including bias, drifts, precision degradation, and loss of considerable amount of data due to operational issues [2]. Missing data is a pervasive issue which occur in most real-world datasets including medical records [3,4], geo-informatics [5], traffic flow [6] and industrial applications [7,8]. The European Union Data Quality Directive (EU-DQD) [9] defined the data quality objective (DQO) that a monitoring method needs to comply with to be used as indicative measurement for regulative purposes.…”

Section: Introductionmentioning

confidence: 99%

Missing Data Imputation on IoT Sensor Networks: Implications for on-site Sensor Calibration

Okafor¹,

Delaney²

2021

Preprint

View full text Add to dashboard Cite

IoT sensors are gaining more popularity in the environmental monitoring space due to their relatively small size, cost of acquisition and ease of installation and operation. They are becoming increasingly important<br>supplement to traditional monitoring systems, particularly for in-situ based monitoring. However, data collection based on IoT sensors are often plagued with missing values usually occurring as a result of sensor faults, network failures, drifts and other operational issues. Several imputation strategies have been proposed for handling missing values in various application domains. This paper examines the performance of different imputation techniques including Multiple Imputation by Chain Equations (MICE), Random forest based imputation (missForest) and K-Nearest Neighbour (KNN) for handling missing values on sensor networks deployed for the quantification of Green House Gases(GHGs). Two tasks were conducted: first, Ozone (O3) and NO2/O3 concentration data collected using Aeroqual and Cairclip sensors respectively over a six months data collection period were corrupted by removing data intervals at different missing periods (p) where p 2 f1day; 1week; 2weeks; 1monthg and also at random points on the dataset at varying proportion (r) where r 2 f5%; 10%; 30%; 50%; 70%g. The missing data were then filled using the different imputation strategies and their imputation accuracy calculated. Second, the performance of sensor calibration by different regression models including Multi Linear Regression (MLR), Decision Tree (DT), Random Forest (RF) and XGBoost (XGB) trained on the different imputed datasets were evaluated. The analysis showed the MICE technique to outperform the others in imputing the missing values on both the O3 and NO2/O3 datasets when missingness was introduced over periods p. MissForest, however, outperformed the rest when missingness was introduced as randomly occuring point errors. While the analysis demonstrated the effects of missing and imputed data on sensor calibration, experimental results showed that a simple model on the imputed dataset can achieve state of-the-art result on in-situ sensor calibration, improving the data quality of the sensor.

show abstract

Treating Missing Data in Industrial Data Analytics

Cited by 23 publications

References 20 publications

Missing Data Patterns: From Theory to an Application in the Steel Industry

Missing Data Patterns: From Theory to an Application in the Steel Industry

Descriptive Statistical Analysis of Frequency control-related variables of Nordic Power System

Missing Data Imputation on IoT Sensor Networks: Implications for on-site Sensor Calibration

Contact Info

Product

Resources

About