A Novel Sensor Data Pre-Processing Methodology for the Internet of Things Using Anomaly Detection and Transfer-By-Subspace-Similarity Transformation

Abstract

The Internet of Things (IoT) and sensors are becoming increasingly popular, especially in monitoring large and ambient environments. Applications that embrace IoT and sensors often require mining the data feeds that are collected at frequent intervals for intelligence. Despite the fact that such sensor data are massive, most of the data contents are identical and repetitive; for example, human traffic in a park at night. Most of the traditional classification algorithms were originally formulated decades ago, and they were not designed to handle such sensor data effectively. Hence, the performance of the learned model is often poor because of the small granularity in classification and the sporadic patterns in the data. To improve the quality of data mining from the IoT data, a new pre-processing methodology based on subspace similarity detection is proposed. Our method can be well integrated with traditional data mining algorithms and anomaly detection methods. The pre-processing method is flexible for handling similar kinds of sensor data that are sporadic in nature that exist in many ambient sensing applications. The proposed methodology is evaluated by extensive experiment with a collection of classical data mining models. An improvement over the precision rate is shown by using the proposed method.

Hybrid Deep-GAN Model for Intrusion Detection in IoT Through Enhanced Whale Optimization

“…In [14], it was recommended fresh IDS by utilizing Transfer by Subspace Similarity classification algorithm that improves the gathered data quality by computing the relevance correlations between variable and filling the missing values in the dataset. Assessment conducted on NSL-KDD database demonstrated that the method is highly successful compared with that which is reliant on Transfer by Subspace Similarity technique.…”

Intelligent Dynamic Real-Time Spectrum Resource Management for Industrial IoT in Edge Computing

“…In general, vast amounts of sensing data from industrial IoT deployed in poor environments are frequently generated by repetitive and irregular data owing to various transmission errors, sensor malfunctions, and cases of external interference, which can eventually lead to poor performance due to incorrect predictions being made in case-based reasoning [ 19 , 20 ]. Therefore, we propose the use of data collection, data preprocessing, and ANN to effectively improve prediction performance through the accurate pattern learning of sensed data [ 21 ].…”

Reinforcement Learning-Based Genetic Algorithm in Optimizing Multidimensional Data Discretization Scheme

“…Choose a state from the state set as the initial state S(t); while t is less than the user's termination iterations do Choose an action from the action set {G, H, I} by e-greedy strategy according to the definition of action in Part C of Section 3; Execute the selected action on the current state S(t) to jump to the next state S(t + 1); Perform crossover operation with global variable on the features contained in state S(t + 1); Calculate the fitness of the multidimensional data discretization scheme after crossover operation using equation 5; Measure the corresponding reward using equation (11) according to the definition of reward in Part C of Section 3; Update crossover Q-Table using equation 6; if the fitness of the multidimensional data discretization scheme > local variable do Update local variable with the fitness of the multidimensional data discretization scheme; end Perform crossover operation in P(t); Calculate the fitness of each individual in P(t) using equation 8; Update global variable with the optimal individual fitness value in P(t); S(t) � S(t + 1); Choose an action from the action set {G, H, I} by e-greedy strategy according to the definition of action in Part C of Section 3; Execute the selected action on the current state S(t) to jump to the next state S(t + 1); Perform mutation operation on the features contained in state S(t + 1) Calculate the fitness of the multidimensional data discretization scheme after mutation operation using equation 5; Measure the corresponding reward using equation (11) according to the definition of reward in Part C of Section 3; Update mutation Q-Table using equation (6); if the fitness of the multidimensional data discretization scheme > local variable do Update local variable with the fitness of the multidimensional data discretization scheme; end Perform mutation operation in P(t); Calculate the fitness of each individual in P(t) using equation (8); Update global variable with the optimal individual fitness value in P(t); t � t + 1; end Return Max(global variable, local variable); end ALGORITHM 2: RLGA algorithm process.…”

“…ese are mainly from various types of sensors, with high-dimensional, incomplete, random, fuzzy, and strong interference and other characteristics [6]. Despite the growing body of artificial intelligence research, how to extract and analyze valuable information from these massive amounts of complex sensor data is still a huge challenge in the field of artificial intelligence [7][8][9]. As one of the most influential data preprocessing technologies, feature discretization can reduce the complexity of data by transforming the continuous features in massive data to discrete features and obtain shorter, more accurate, and more comprehensible rules, so as to improve the efficiency of data mining and machine learning [10][11][12][13][14][15][16].…”