In the realm of managing malignant liver tumors, the convergence of radiomics and machine learning has redefined the landscape of medical practice. The field of radiomics employs advanced algorithms to extract thousands of quantitative features (including intensity, texture, and structure) from medical images. Machine learning, including its subset deep learning, aids in the comprehensive analysis and integration of these features from diverse image sources. This potent synergy enables the prediction of responses of malignant liver tumors to various treatments and outcomes. In this comprehensive review, we examine the evolution of the field of radiomics and its procedural framework. Furthermore, the applications of radiomics combined with machine learning in the context of personalized treatment for malignant liver tumors are outlined in aspects of surgical therapy and non-surgical treatments such as ablation, transarterial chemoembolization, radiotherapy, and systemic therapies. Finally, we discuss the current challenges in the amalgamation of radiomics and machine learning in the study of malignant liver tumors and explore future opportunities.