Anomaly Detection with Machine Learning Algorithms and Big Data in Electricity Consumption

Abstract

When analyzing smart metering data, both reading errors and frauds can be identified. The purpose of this analysis is to alert the utility companies to suspicious consumption behavior that could be further investigated with on-site inspections or other methods. The use of Machine Learning (ML) algorithms to analyze consumption readings can lead to the identification of malfunctions, cyberattacks interrupting measurements, or physical tampering with smart meters. Fraud detection is one of the classical anomaly detection examples, as it is not easy to label consumption or transactional data. Furthermore, frauds differ in nature, and learning is not always possible. In this paper, we analyze large datasets of readings provided by smart meters installed in a trial study in Ireland by applying a hybrid approach. More precisely, we propose an unsupervised ML technique to detect anomalous values in the time series, establish a threshold for the percentage of anomalous readings from the total readings, and then label that time series as suspicious or not. Initially, we propose two types of algorithms for anomaly detection for unlabeled data: Spectral Residual-Convolutional Neural Network (SR-CNN) and an anomaly trained model based on martingales for determining variations in time-series data streams. Then, the Two-Class Boosted Decision Tree and Fisher Linear Discriminant analysis are applied on the previously processed dataset. By training the model, we obtain the required capabilities of detecting suspicious consumers proved by an accuracy of 90%, precision score of 0.875, and F1 score of 0.894.

Feature engineering solution with structured query language analytic functions in detecting electricity frauds using machine learning

“…Our proposed approach can be implemented for similar datasets measured by conventional meters, but the SQL analytic functions are appropriate for time series generated by smart meters as well. For smart meter-recorded data, there are other approaches that involve specific machine leaning algorithms and time series feature extraction library for electricity consumption fraud detection in smart grids 7 . However, although the trend is to gradually replace the conventional meters, there are still large regions even in the developed countries that continue to measure electricity with conventional meters.…”

“… Ref. Data provider Meter type Algorithm Skewness Data processing Feature engineering 1 State Grid Corporation of China Smart meter Three-sigma Rule, LR, RF, SVM, wide and deep convolutional neural network (CNN) Not approached Interpolation max–min scaling No 2 Brazilian electric utility Smart meter Optimum-path forest classifier Not approached Feature selection with black hole algorithm No 4 Brazilian electric utility, CPFL Energia Smart meter CNN Not approached Time series and image processing Image feature extractors 5 Irish Commission for Energy Regulation (CER) Smart meter RF, KNN, SVM, NN, gradient boosting machine (GBM) classification model Not necessary as there is no fraud indication Finite mixture model clustering for customer segmentation Genetic programming algorithm 6 Irish CER Smart meter SVM Not necessary Clustering No 7 Irish CER Smart meter Spectral residual-convolutional neural network and an anomaly trained model based on martingales Not necessary Data transformation, feature extraction, Fisher discriminant analysis No ...…”

“…In many cases, the target data is missing, therefore the problem becomes an unsupervised one and could be solved with anomaly detection algorithms 7 . The difference between logistic regression, decision tree or other classifiers and anomaly detection algorithms is the existence of the target in the training set.…”

Smart Cities and Awareness of Sustainable Communities Related to Demand Response Programs: Data Processing with First-Order and Hierarchical Confirmatory Factor Analyses

“…Consumption data and questionnaires deployed by the Irish CER were involved in obtaining a clustering solution [23], the classification of load profiles [24,25], extracting insights from smart metering data and responses of electricity consumers [26], anomaly detection [27], and forecasts [28][29][30].…”

Entropy-Based Anomaly Detection in Household Electricity Consumption

“…It could be used to improve the privacy and security of residents. In addition, if there has been no change in the pattern of user behavior, this information can be used for the identification of an appliance failure, a sudden change in the charging system, or fraud in the resident's electrical system [38].…”