A big data cleaning method based on improved CLOF and Random Forest for distribution network

Liu, Jie; Cao, Yijia; Guo, Yixiu; Deng, Wei

doi:10.17775/cseejpes.2020.04080

Cited by 6 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The results achieved by the proposed method were presented in figures. Liu et al [29] improved the cluster-based local outlier factor (CLOF) and the random forest algorithm to impute missing data and detect outliers in batch processing. Datasets comprising data on electricity consumption, network loss rate, and active power were utilized to compare the results with those achieved by the old CLOF algorithm in terms of detecting outliers, and they were also compared to K-nearest neighbors, MICE, and missForest results in terms of imputing missing values.…”

Section: Machine Learningmentioning

confidence: 99%

“…Furthermore, both Fitters et al [40] and Albattah et al [45] detected contextual outliers without tackling missing data. In addition, Liu et al [29] and Kulanuwat et al [54] both dealt with outliers before dealing with the missing value. However, missing values need to be tackled before detecting outliers, especially contextual outliers, because missing values may hide outliers and affect outlier detection Han et al [3].…”

Section: Rq2: Which Data Cleaning Issue Is Most Commonly Discussed Du...mentioning

confidence: 99%

See 1 more Smart Citation

Cleaning Big Data Streams: A Systematic Literature Review

2023

View full text Add to dashboard Cite

In today’s big data era, cleaning big data streams has become a challenging task because of the different formats of big data and the massive amount of big data which is being generated. Many studies have proposed different techniques to overcome these challenges, such as cleaning big data in real time. This systematic literature review presents recently developed techniques that have been used for the cleaning process and for each data cleaning issue. Following the PRISMA framework, four databases are searched, namely IEEE Xplore, ACM Library, Scopus, and Science Direct, to select relevant studies. After selecting the relevant studies, we identify the techniques that have been utilized to clean big data streams and the evaluation methods that have been used to examine their efficiency. Also, we define the cleaning issues that may appear during the cleaning process, namely missing values, duplicated data, outliers, and irrelevant data. Based on our study, the future directions of cleaning big data streams are identified.

show abstract

Section: Machine Learningmentioning

confidence: 99%

Section: Rq2: Which Data Cleaning Issue Is Most Commonly Discussed Du...mentioning

confidence: 99%

Cleaning Big Data Streams: A Systematic Literature Review

2023

View full text Add to dashboard Cite

show abstract

“…After that, the missing values labelled with "−2" and the normalized data are combined to form the dataset. Choosing −2 is to distinguish missing values from the normalized data (Liu et al, 2020). In the second stage, we randomly sample from the dataset to train PCA-KMeans, GMM, and iForest algorithms to predict outliers using a soft voting mechanism.…”

Section: Problem Descriptionsmentioning

confidence: 99%

A Flexible Ensemble Algorithm for Big Data Cleaning of PMUs

Long¹,

He²,

Liu³

et al. 2021

Front. Energy Res.

View full text Add to dashboard Cite

With an increasing application of Phase Measurement Units in the smart grid, it is becoming inevitable for PMUs to operate in severe conditions, which results in outliers and missing data. However, conventional techniques take excessive time to clean outliers and fill missing data due to lacking support from a big data platform. In this paper, a flexible ensemble algorithm is proposed to implement a precise and scalable data clean by the existing big data platform “Apache Spark.” In the proposed scheme, an ensemble model based on a soft voting approach utilizes principal component analysis in conjunction with the K-means, Gaussian mixture model, and isolation forest technique to detect outliers. The proposed scheme uses a gradient boosting decision tree for each extracted feature of PMUs for the data filling process after detecting outliers. The test results demonstrate that the proposed model achieves high accuracy and recall by comparing simulated and real-world Phase measurement unit data using the local outlier factor algorithm and Density-Based Spatial Clustering of Application with Noise (DBSCAN). The mean absolute error, root mean square error and R2-score criteria are used to validate the proposed method’s data filling results against contemporary techniques such as decision tree and linear regression algorithms.

show abstract

“…Furthermore, ref. [131] has implemented local outlier factor and random forest to clean big data related to distribution networks. As another example, an abnormal data cleaning approach related to wind turbines has been introduced in [132].…”

mentioning

confidence: 99%

Deep Learning for Forecasting-Based Applications in Cyber–Physical Microgrids: Recent Advances and Future Directions

et al. 2023

View full text Add to dashboard Cite

Renewable energy resources can be deployed locally and efficiently using the concept of microgrids. Due to the natural uncertainty of the output power of renewable energy resources, the planning for a proper operation of microgrids can be a challenging task. In addition, the information about the loads and the power consumption of them can create benefits to increase the efficiency of the microgrids. However, electrical loads can have uncertainty due to reasons such as unpredictable behavior of the consumers. To exploit a microgrid, energy management is required at the upper level of operation and control in order to reduce the costs. One of the most important tasks of the energy management system is to satisfy the loads and, in other words, develop a plan to maintain equilibrium between the power generation and power consumption. To obtain information about the output power of renewable energy resources and power consumption, deep learning can be implemented as a powerful tool, which is able to predict the desired values. In addition, weather conditions can affect the output power of renewable energy-based resources and the behavior of the consumers and, as a result, the power consumption. So, deep learning can be deployed for the anticipation of the weather conditions. This paper will study the recent works related to deep learning, which has been implemented for the prediction of the output power of renewable energy resources (i.e., PVs and wind turbines), electrical loads, and weather conditions (i.e., solar irradiance and wind speed). In addition, for possible future directions some strategies are suggested, the most important of which is the implementation of quantum computing in cyber–physical microgrids.

show abstract

A big data cleaning method based on improved CLOF and Random Forest for distribution network

Cited by 6 publications

References 0 publications

Cleaning Big Data Streams: A Systematic Literature Review

Cleaning Big Data Streams: A Systematic Literature Review

A Flexible Ensemble Algorithm for Big Data Cleaning of PMUs

Deep Learning for Forecasting-Based Applications in Cyber–Physical Microgrids: Recent Advances and Future Directions

Contact Info

Product

Resources

About