Medical imaging plays an indispensable role in evaluating, predicting, and monitoring a range of medical conditions. Radiomics, a specialized branch of medical imaging, utilizes quantitative features extracted from medical images to describe underlying pathologies, genetic information, and prognostic indicators. The integration of radiomics with artificial intelligence presents innovative avenues for cancer diagnosis, prognosis evaluation, and therapeutic choices. In the context of oncology, radiomics offers significant potential. Feature selection emerges as a pivotal step, enhancing the clinical utility and precision of radiomics. It achieves this by purging superfluous and unrelated features, thereby augmenting model performance and generalizability. The goal of this review is to assess the fundamental radiomics process and the progress of feature selection methods, explore their applications and challenges in cancer research, and provide theoretical and methodological support for future investigations. Through an extensive literature survey, articles pertinent to radiomics and feature selection were garnered, synthesized, and appraised. The paper provides detailed descriptions of how radiomics is applied and challenged in different cancer types and their various stages. The review also offers comparative insights into various feature selection strategies, including filtering, packing, and embedding methodologies. Conclusively, the paper broaches the limitations and prospective trajectories of radiomics.