Compact DINO-ViT: Feature Reduction for Visual Transformer

Abstract

Research has been ongoing for years to discover image features that enable their best classification. One of the latest developments in this area is the Self-Distillation with No Labels Vision Transformer—DINO-ViT features. However, even for a single image, their volume is significant. Therefore, for this article we proposed to substantially reduce their size, using two methods: Principal Component Analysis and Neighborhood Component Analysis. Our developed methods, PCA-DINO and NCA-DINO, showed a significant reduction in the volume of the features, often exceeding an order of magnitude while maintaining or slightly reducing the classification accuracy, which was confirmed by numerous experiments. Additionally, we evaluated the Uniform Manifold Approximation and Projection (UMAP) method, showing the superiority of the PCA and NCA approaches. Our experiments involving modifications to patch size, attention heads, and noise insertion in DINO-ViT demonstrated that both PCA-DINO and NCA-DINO exhibited reliable accuracy. While NCA-DINO is optimal for high-performance applications despite its higher computational cost, PCA-DINO offers a faster, more resource-efficient solution, depending on the application-specific requirements. The code for our method is available on GitHub.