Large dataset summarization with automatic parameter optimization and parallel processing for local outlier detection

Abstract

Summary As one of the most important research problems of data analytics and data mining, outlier detection from large datasets has drawn many research attentions in recent years. In this paper, we investigate the interesting research problem of summarizing large datasets for supporting efficient local outlier detection. To summarize large datasets, efficient summarization algorithms are proposed that produce a highly compact summary of the original dataset, which can be applied to detect local outliers from future similar datasets. A novel automatic parameter optimization method is proposed to produce the optimal setup of the key parameters used in the summarization algorithm. Parallel processing methods are also proposed to accelerate significantly the summarization process. The experimental evaluation results demonstrate that our proposed algorithms are highly scalable for large datasets and effective in producing dataset summary for local outlier detection.

Expecting individuals’ body reaction to Covid-19 based on statistical Naïve Bayes technique

“…The proposed HOR method mainly composes of two stages, called; Fast Rejection (FR) stage and Accurate Rejection (AR) stage as shown in figure 5 . In FR stage, standard division is used as a statistical-based method to quickly reject outliers from the training dataset as possible [ 21 , 22 ]. In AR stage, Binary Gray Wolf Optimization (BGWO) method is used as an optimization technique to accurately remove the rest of outliers in the training data to improve the performance of the classification model [23] .…”

High performance deep learning techniques for big data analytics

“…Similarly, Zhang et al apply Hadoop in speeding up the computation in power load forecast by clustering the spatial‐temporal load data. Shou and Li parallelize the process of local outlier detection in large data set summarization albeit Hadoop is not employed in this work.…”