An Electric Power Sensor Data Oriented Data Cleaning Solution

Li, He; Chen, Jiangqi; Huang, Fupeng; Li, Han

doi:10.1109/ispan-fcst-iscc.2017.29

Cited by 4 publications

(2 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Outliers, also known as anomalies [82] and spikes [36,83], are values that exceed thresholds or largely deviate from the normal behaviour provided by the model. A sensor data measurement is also considered an outlier if it is significantly different from its previous and next observations or observations from neighbouring sensor nodes [38,45,48]. Outliers are also known as faults, though faults also include other types of errors such as bias, drifts, noise, constant value, and stuck-at-zero.…”

Section: Types Of Errors In Sensor Datamentioning

confidence: 99%

See 1 more Smart Citation

Sensor data quality: a systematic review

2020

View full text Add to dashboard Cite

Sensor data quality plays a vital role in Internet of Things (IoT) applications as they are rendered useless if the data quality is bad. This systematic review aims to provide an introduction and guide for researchers who are interested in quality-related issues of physical sensor data. The process and results of the systematic review are presented which aims to answer the following research questions: what are the different types of physical sensor data errors, how to quantify or detect those errors, how to correct them and what domains are the solutions in. Out of 6970 literatures obtained from three databases (ACM Digital Library, IEEE Xplore and ScienceDirect) using the search string refined via topic modelling, 57 publications were selected and examined. Results show that the different types of sensor data errors addressed by those papers are mostly missing data and faults e.g. outliers, bias and drift. The most common solutions for error detection are based on principal component analysis (PCA) and artificial neural network (ANN) which accounts for about 40% of all error detection papers found in the study. Similarly, for fault correction, PCA and ANN are among the most common, along with Bayesian Networks. Missing values on the other hand, are mostly imputed using Association Rule Mining. Other techniques include hybrid solutions that combine several data science methods to detect and correct the errors. Through this systematic review, it is found that the methods proposed to solve physical sensor data errors cannot be directly compared due to the non-uniform evaluation process and the high use of non-publicly available datasets. Bayesian data analysis done on the 57 selected publications also suggests that publications using publicly available datasets for method evaluation have higher citation rates.

show abstract

Section: Types Of Errors In Sensor Datamentioning

confidence: 99%

“…Liu et al [45] presented another example of the clustering method used for outlier detection using Time-Relevant k-Means clustering for electric power sensor data. The k-means clustering algorithm is used to form initial clusters.…”

Section: Support Vector Machine a Support Vector Machine (Svm) Is A Mmentioning

confidence: 99%