Is big data sufficient for a reliable detection of non-technical losses?

Abstract

Non-technical losses (NTL) occur during the distribution of electricity in power grids and include, but are not limited to, electricity theft and faulty meters. In emerging countries, they may range up to 40% of the total electricity distributed. In order to detect NTLs, machine learning methods are used that learn irregular consumption patterns from customer data and inspection results. The Big Data paradigm followed in modern machine learning reflects the desire of deriving better conclusions from simply analyzing more data, without the necessity of looking at theory and models. However, the sample of inspected customers may be biased, i.e. it does not represent the population of all customers. As a consequence, machine learning models trained on these inspection results are biased as well and therefore lead to unreliable predictions of whether customers cause NTL or not. In machine learning, this issue is called covariate shift and has not been addressed in the literature on NTL detection yet. In this work, we present a novel framework for quantifying and visualizing covariate shift. We apply it to a commercial data set from Brazil that consists of 3.6M customers and 820K inspection results. We show that some features have a stronger covariate shift than others, making predictions less reliable. In particular, previous inspections were focused on certain neighborhoods or customer classes and that they were not sufficiently spread among the population of customers. This framework is about to be deployed in a commercial product for NTL detection.

On the reduction of biases in big data sets for the detection of irregular power usage

“…There may be also covariate shifts for other master data, such as for the customer class or for the contract status. 2 We now aim to correct n different biases at a same time, e.g. for class imbalance as well as different types of covariate shift.…”

“…We have previously shown that in many cases, the set of inspected customers is biased. 2 A reason for that is that past inspections have been largely focused on certain criteria and were not sufficiently spread across the population. This paper builds on top of our previous contributions and aims at bias reduction in data, and further at more generalizable NTL predictors.…”

“…Covariate shift has been recognized as an issue in statistics since the mid-20th century. 2 In contrast, it has received only a limited attention in machine learning, mainly within the computational learning theory subfield, yet the situation is currently evolving. 3,4 Training Test Non-technical losses (NTL) appear in power grids during distribution and include, but are not limited to, the following causes: meter tampering in order to record lower consumptions, bypassing meters by rigging lines from the power source, arranged false meter readings by bribing meter readers or faulty or broken meters.…”

“…We have previously shown that in many cases, the set of inspected customers is biased. 2 A reason for that is that past inspections have been largely focused on certain criteria and were not sufficiently spread across the population.…”

Identifying Irregular Power Usage by Turning Predictions into Holographic Spatial Visualizations

“…the set of customers to generate inspections for) having different distributions. We have shown in [8] that the sample of inspected customers may be biased, i.e. it does not represent the population of all customers.…”