Dimensionality reduction using singular vectors

Abstract

A common problem in machine learning and pattern recognition is the process of identifying the most relevant features, specifically in dealing with high-dimensional datasets in bioinformatics. In this paper, we propose a new feature selection method, called Singular-Vectors Feature Selection (SVFS). Let $$D= [A \mid \mathbf {b}]$$ D = [ A ∣ b ] be a labeled dataset, where $$\mathbf {b}$$ b is the class label and features (attributes) are columns of matrix A. We show that the signature matrix $$S_A=I-A^{\dagger }A$$ S A = I - A † A can be used to partition the columns of A into clusters so that columns in a cluster correlate only with the columns in the same cluster. In the first step, SVFS uses the signature matrix $$S_D$$ S D of D to find the cluster that contains $$\mathbf {b}$$ b . We reduce the size of A by discarding features in the other clusters as irrelevant features. In the next step, SVFS uses the signature matrix $$S_A$$ S A of reduced A to partition the remaining features into clusters and choose the most important features from each cluster. Even though SVFS works perfectly on synthetic datasets, comprehensive experiments on real world benchmark and genomic datasets shows that SVFS exhibits overall superior performance compared to the state-of-the-art feature selection methods in terms of accuracy, running time, and memory usage. A Python implementation of SVFS along with the datasets used in this paper are available at https://github.com/Majid1292/SVFS.

Absorption Spectroscopy in Dental Tissue Analysis

“…During the processing stage, the highest value of this criterion was found for features f 1 and f 2, which represent the wavelength value at maximum reflectance and the initial gradient value of the observed spectrum. Standard methods of identifying the most important features are often based on singular value decomposition [37], QR factorization with Gram-Schmidt orthogonalization process, and principal component analysis [38].…”

Machine learning-based meta-analysis of colorectal cancer and inflammatory bowel disease

“…However, due to the computational complexity, wrapper methods are not feasible for high-dimensional datasets like those employed in this paper. In this research, we utilized SVFS (Singular Vectors Feature Selection), a hybrid feature selection method that has recently demonstrated superior results compared to other methods on gene expression data [30]. SVFS is a method designed for high-dimensional datasets.…”

“…SVFS is a method designed for high-dimensional datasets. Given a matrix A with its Moore-Penrose pseudo-inverse A † , it is shown in [30,31] that the projector P A = I − A † A partitions features into clusters based on their correlations. Initially, SVFS identifies and retains only those features that correlate with the class label, discarding others as irrelevant.…”

“…We incorporated the parameters suggested for biological data in the paper as parameters for our algorithm (Th irr = 3, Th red = 4, α = 50, β = 5, and k = 100) [30]. Executing the SVFS feature selection algorithm may yield varying subsets of features on each iteration.…”

Semi-Supervised Dimensionality Reduction Method Utilizing Pairwise Constraints and Integrating Similarity and Dissimilarity among Data

“…Direct computation of high-dimensional data not only leads to low computational efficiency but can also trigger the "curse of dimensionality" 4 , a phenomenon where the performance of algorithms drastically declines as the dimensionality of data increases. In addition to feature selection 5 , dimensionality reduction techniques 6 have become a key solution to this problem, aiming to effectively reduce the dimensionality of data while preserving as much of the original information and structural characteristics as possible.…”