Vehicle‐mounted detection methods have been widely applied in the maintenance of high‐speed railways (HSRs), providing feasibility for diagnosing ballastless track arching. However, applying detection data faces several key limitations: (1) The threshold mostly requires manual setting, making recognition accuracy highly subjective; (2) the extensive workload of manual inspections makes it challenging to label detection data, hindering the application of supervised learning approaches. To address these problems, this paper utilizes the longitudinal level irregularity data obtained from vehicle‐mounted detection, employing the concept of unsupervised learning for dimensionality reduction, combined with clustering algorithms and minimal label fine‐tuning, to design two frameworks: the fully unsupervised framework (FUF) and the few‐shot fine‐tuned framework (FFF). Experiments on dynamic detection data from a Chinese HSR line were conducted, comparing the performance of data dimensionality reduction, clustering, and classification under different strategy combinations. The results show that the improved variational autoencoder significantly enhances the performance of the encoder in dimensionality reduction, facilitating better feature extraction; the FUF achieves effective clustering outcomes without any labeled samples and its adjusted rand index score exceeded 0.8, showcasing its robustness and applicability in scenarios with no prior annotations; the FFF requires only a small number of labeled samples (labeling ratio of 5%) and achieves excellent performance, with metrics such as accuracy exceeding 0.85, thus greatly reducing the reliance on labeled data. This study offers a novel method for solving engineering issues with limited labeled data, providing an efficient solution for identifying track arching defects and advancing railway infrastructure monitoring.