PAS3-HSID: a Dynamic Bio-Inspired Approach for Real-Time Hot Spot Identification in Data Streams

Abstract

Hot spot identification is a very relevant problem in a wide variety of areas such as health care, energy or transportation. A hot spot is defined as a region of high likelihood of occurrence of a particular event. To identify hot spots, location data for those events is required, which is typically collected by telematics devices. These sensors are constantly gathering information, generating very large volumes of data. Current state-of-the-art solutions are capable of identifying hot spots from big static batches of data by means of variations of clustering or instance selection techniques that pre-process the original input data, providing the most relevant locations. However, these approaches neglect to address changes in hot spots over time.Method: This paper presents a dynamic bio-inspired approach to detect hot spots in big data streams. This computational intelligence method is designed and applied to the transportation sector as a case study to identify incidents in the roads caused by heavy goods vehicles. We adapt an immune-based algorithm to account for the temporary aspect of hot spots inspired by the idea of pheromones, which is then subsequently implemented using Apache Spark Streaming.Results: Experimental results on real datasets with up to 4.5 million data points -provided by a telematics company -show that the algorithm is capable of quickly processing large streaming batches of data, as well as successfully adapting over time to detect hot spots.

Fuzzy Hot Spot Identification for Big Data: An Initial Approach

“…The shortcomings of these methods lie in the fact that they may produce invalid results, require historical data or a predefined number of hot spots, and are not suitable for big data processing. Detailed reviews of the existing methods for HSID can be found in [5], [8]. The literature on employing concepts of fuzzy logic for HSID is limited.…”

“…In this subsection we provide a high-level description of the streaming SeleSup-HSID algorithm, based on the earlier works on SeleSup-HSID in [5] and [6], and used here as the method into which we introduce fuzzy concepts. A full description can be found in [8] and our implementation is available on GitHub 1 (using Apache Spark Streaming [14]). The method is designed to work on a large data stream of incident data, processed as a sequential chain of microbatches containing incidents from a given time interval.…”

“…We compare the fuzzy approach with the non-fuzzy streaming SeleSup-HSID algorithm, both implemented under Apache Spark Streaming [14], a big data processing framework. For the non-fuzzy version, we use dr=0.3, delTh=1.9, hsTh=3 and mileage ranges of 0.5 and 5.0 for harsh braking and contextual speeding data respectively, as was done in [8]. For the fuzzy method, we use maxMiles=2, dr=0.3, delTh=1.2 and hsTh=3.…”

“…In Figure 3, the regular drops in the percentage of dense hot spots coincide with data from weekends. As discussed in [8], at these times there are fewer HGVs active, and therefore a reduction in the number of incidents occurring. Over the weekends, dense hot spots decay into sparse hot spots, resulting in the patterns seen in this figure . Overall, these results demonstrate that this initial fuzzy approach, despite utilising simple techniques, is able to provide more precise and informative hot spots, with a minor increase in the processing time.…”

“…Experimental results showed the success of this method (SeleSup-HSID) as a HSID approach. This approach was later extended [8], adapting the methodology in order to make it suitable for processing large, dynamic data streams.…”