Dimensionality reduction by feature clustering for regression problems

Xu, Rong-Fang; Lee, Shie-Jue

doi:10.1016/j.ins.2014.12.003

Cited by 30 publications

(8 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…3) PALM features a fully open network structure where its rules can be automatically generated, merged and updated on demand in the one-pass learning fashion. The rule generation process is based on the self-constructing clustering approach [23], [24] checking coherence of input and output space. The rule merging scenario is driven by the similarity analysis via the distance and orientation of two hyperplanes.…”

Section: Introductionmentioning

confidence: 99%

PALM: An Incremental Construction of Hyperplanes for Data Stream Regression

Ferdaus

Pratama

Anavatti

et al. 2019

IEEE Trans. Fuzzy Syst.

View full text Add to dashboard Cite

Data stream has been the underlying challenge in the age of big data because it calls for real-time data processing with the absence of a retraining process and/or an iterative learning approach. In realm of fuzzy system community, data stream is handled by algorithmic development of self-adaptive neurofuzzy systems (SANFS) characterized by the single-pass learning mode and the open structure property which enables effective handling of fast and rapidly changing natures of data streams. The underlying bottleneck of SANFSs lies in its design principle which involves a high number of free parameters (rule premise and rule consequent) to be adapted in the training process. This figure can even double in the case of type-2 fuzzy system. In this work, a novel SANFS, namely parsimonious learning machine (PALM), is proposed. PALM features utilization of a new type of fuzzy rule based on the concept of hyperplane clustering which significantly reduces the number of network parameters because it has no rule premise parameters. PALM is proposed in both type-1 and type-2 fuzzy systems where all of which characterize a fully dynamic rule-based system. That is, it is capable of automatically generating, merging and tuning the hyperplane-based fuzzy rule in the single pass manner. Moreover, an extension of PALM, namely recurrent PALM (rPALM), is proposed and adopts the concept of teacher-forcing mechanism in the deep learning literature. The efficacy of PALM has been evaluated through numerical study with six real-world and synthetic data streams from public database and our own real-world project of autonomous vehicles. The proposed model showcases significant improvements in terms of computational complexity and number of required parameters against several renowned SANFSs, while attaining comparable and often better predictive accuracy.

show abstract

Section: Introductionmentioning

confidence: 99%

PALM: An Incremental Construction of Hyperplanes for Data Stream Regression

Ferdaus

Pratama

Anavatti

et al. 2019

IEEE Trans. Fuzzy Syst.

View full text Add to dashboard Cite

show abstract

“…Intuitively, similar instances are to be grouped in a cluster and different instances grouped in different clusters. Clustering has been widely utilized in a variety of applications, such as revealing the internal structure of the data [3], deriving segmentation of the data [4,5], preprocessing the data for other artificial intelligence (AI) techniques [6,7], business intelligence [1,8], and knowledge discovery in data [9,10]. For example, in electronic text processing [11][12][13], clustering is used to reduce the dimensionality in order to improve the efficiency of the processing, or to ease the curse of dimensionality encountered for high-dimensional problems.…”

Section: Introductionmentioning

confidence: 99%

“…• The feeding order of x 9 , x 10 , x 4 , x 7 , x 5 , x 11 , x 2 , x 12 , x 3 , x 1 , x 6 , x 8 . After performing SCC in the first iteration, there are 5 clusters: C 1 , C 2 , C 3 , C 4 , and C 5 , as shown in Figure A2a Instances x 1 , x 5 , x 8 , and x 12 are assigned to C 1 , x 2 and x 10 are assigned to C 2 , x 3 and x 6 are assigned to C 3 , x 4 , x 7 , and x 11 are assigned to C 4 , and x 9 is assigned to C 5 . Iterations 2 and 3 are performed subsequently.…”

mentioning

confidence: 99%

“…Iterations 2 and 3 are performed subsequently. After the fourth iteration, we have 3 clusters: C 1 , C 2 , and C 3 , as shown in Figure A2b Instances x 1 , x 5 , x 8 , and x 12 are assigned to C 1 , x 2 , x 3 , x 6 , x 9 , and x 10 are assigned to C 2 , and x 4 , x 7 , and x 11 are assigned to C 3 . Then the cluster assignments are stable and so SCC-I stops with three clusters C 1 , C 2 , and C 3 as shown above, with 4, 5, and 3 instances assigned to them, respectively.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Weighted z-Distance-Based Clustering and Its Application to Time-Series Data

Wang

Lin

et al. 2019

Applied Sciences

Self Cite

View full text Add to dashboard Cite

Clustering is the practice of dividing given data into similar groups and is one of the most widely used methods for unsupervised learning. Lee and Ouyang proposed a self-constructing clustering (SCC) method in which the similarity threshold, instead of the number of clusters, is specified in advance by the user. For a given set of instances, SCC performs only one training cycle on those instances. Once an instance has been assigned to a cluster, the assignment will not be changed afterwards. The clusters produced may depend on the order in which the instances are considered, and assignment errors are more likely to occur. Also, all dimensions are equally weighted, which may not be suitable in certain applications, e.g., time-series clustering. In this paper, improvements are proposed. Two or more training cycles on the instances are performed. An instance can be re-assigned to another cluster in each cycle. In this way, the clusters produced are less likely to be affected by the feeding order of the instances. Also, each dimension of the input can be weighted differently in the clustering process. The values of the weights are adaptively learned from the data. A number of experiments with real-world benchmark datasets are conducted and the results are shown to demonstrate the effectiveness of the proposed ideas.Many types of clustering algorithms have been proposed [2,17]. Similarity or distance measures are core components to cluster similar data into the same clusters, while dissimilar or distant data are placed into different clusters [18]. Centroid-based clustering [19][20][21][22][23][24][25][26] groups data instances in an exclusive way. If an instance belongs to a definite cluster, it could not be included in another cluster. K-means is one such algorithm, well-known in the AI community. To use it, the user has to provide the desired number of clusters, K, in advance. Each instance is assigned to the nearest cluster center. Then the K cluster centers are re-estimated. This process is repeated until the cluster centers are stable. Self-organizing mapping (SOM) employs a set of representatives. When a vector is presented, all representatives compete with each other. The winner is updated so as to move toward the vector. Hierarchical clustering [27][28][29][30][31][32] creates a hierarchical decomposition of the set of data instances using some criteria. Two strategies, bottom-up and top-down, are adopted in hierarchical algorithms. The user usually has to decide how many and what clusters are most desirable from the offered hierarchy of clusters. Distribution-based clustering [33][34][35][36][37][38] is based on distribution models. Fuzzy C-means uses fuzzy sets to cluster instances, and data instances are bound to each cluster by means of a membership function. Therefore, each instance may belong to several clusters with different degrees of membership. Gaussian mixture model-Expectation maximization (GMM-EM) uses a completely probabilistic approach. Each cluster is mathematically represented by a parametr...

show abstract

“…In this step, Correlation Coefficient (CC) is used to measure the redundancy level between features because it can be directly applied to numerical data. Although CC can only measure the linear relationship between variables, it has been shown to be effective in many feature selection methods [99,246]. The CC measure gives a value between -1 and 1 whose absolute value represents the correlation level between two features.…”

Section: Redundancy Based Feature Clustering: Rfcmentioning

confidence: 99%

Evolutionary Computation for Feature Manipulation in Classification on High-dimensional Data

Tran¹

View full text Add to dashboard Cite

<p>More and more high-dimensional data appears in machine learning, especially in classification tasks. With thousands of features, these datasets bring challenges to learning algorithms not only because of the curse of dimensionality but also the existence of many irrelevant and redundant features. Therefore, feature selection and feature construction (or feature manipulation in short) are essential techniques in preprocessing these datasets. While feature selection aims to select relevant features, feature construction constructs high-level features from the original ones to better represent the target concept. Both methods can decrease the dimensionality and improve the performance of learning algorithms in terms of classification accuracy and computation time. Although feature manipulation has been studied for decades, the task on high-dimensional data is still challenging due to the huge search space. Existing methods usually face the problem of stagnation in local optima and/or require high computation time. Evolutionary computation techniques are well-known for their global search. Particle swarm optimisation (PSO) and genetic programming (GP) have shown promise in feature selection and feature construction, respectively. However, the use of these techniques to high-dimensional data usually requires high memory and computation time. The overall goal of this thesis is to investigate new approaches to using PSO for feature selection and GP for feature construction on high-dimensional classification problems. This thesis focuses on incorporating a variety of strategies into the evolutionary process and developing new PSO and GP representations to improve the effectiveness and efficiency of PSO and GP for feature manipulation on high-dimensional data. This thesis proposes a new PSO based feature selection approach to high-dimensional data by incorporating a new local search to balance global and local search of PSO. A hybrid of wrapper and filter evaluation method which can be sped up in the local search is proposed to help PSO achieve better performance, scalability and robustness on high-dimensional data. The results show that the proposed method significantly outperforms the compared methods in 80% of the cases with an increase up to 16% average accuracy while reduces the number of features from one to two orders of magnitude. This thesis develops the first PSO based feature selection via discretisation method that performs both multivariate discretisation and feature selection in a single stage to achieve better solutions than applying these techniques separately in two stages. Two new PSO representations are proposed to evolve cut-points for multiple features simultaneously. The results show that the proposed method selects less than 4.6% of the features in all cases to improve the classification performance from 5% to 23% in most cases. This thesis proposes the first clustering-based feature construction method to improve the performance of single-tree GP on high-dimensional data. A new feature clustering method is proposed to automatically group similar features into the same group based on a given redundancy level. The results show that compared with standard GP, the new method can select less than half of the features to construct a new high-level feature that achieves significantly better accuracy in most cases. The combination of the single constructed feature and the selected ones achieves the best performance among different feature sets created from a single tree. This thesis develops the first class-dependent multiple feature construction method using multi-tree GP for high-dimensional data. A new GP representation and a new filter fitness function that combines two filter measures are proposed to evaluate the whole set of constructed features more effectively and efficiently. The results show that in 83% of the cases, with less than 10 constructed features, the class-dependent method increases up to 32% average accuracy on using all the original thousands of features and 10% on using those constructed by the class-independent method.</p>

show abstract

Dimensionality reduction by feature clustering for regression problems

Cited by 30 publications

References 39 publications

PALM: An Incremental Construction of Hyperplanes for Data Stream Regression

PALM: An Incremental Construction of Hyperplanes for Data Stream Regression

Weighted z-Distance-Based Clustering and Its Application to Time-Series Data

Evolutionary Computation for Feature Manipulation in Classification on High-dimensional Data

Contact Info

Product

Resources

About