BackgroundLiver cancer is one of the major diseases threatening human life and health, and this study aims to explore new methods for treating liver cancer.MethodsA deep learning model for the efficacy of clinical herbal medicines for liver cancer was constructed based on NDCNN, combined with the natural evolutionary rules of a genetic algorithm to obtain the herbal compound for liver cancer treatment. We obtained differential genes between liver cancer tissues and normal tissues from the analysis of TCGA database, screened the active ingredients and corresponding targets of the herbal compound using the TCMSP database, mapped the intersection to obtain the potential targets of the herbal compound for liver cancer treatment in the Venny platform, constructed a PPI network, and conducted GO analysis and KEGG analysis on the targets of the herbal compound for liver cancer treatment. Finally, the key active ingredients and important targets were molecularly docked.ResultsThe accuracy of the NDCNN training set was 0.92, and the accuracy of the test set was 0.84. After combining with the genetic algorithm for 1,000 iterations, a set of Chinese herbal compound prescriptions was finally the output. A total of 86 targets of the herbal compound for liver cancer were obtained, mainly five core targets of IL-6, ESR1, JUN, IL1β, and MMP9. Among them, quercetin, kaempferol, and stigmasterol may be the key active ingredients in hepatocellular carcinoma, and the herbal compound may be participating in an inflammatory response and the immune regulation process by mediating the IL-17 signaling pathway, the TNF signaling pathway, and so on. The anticancer effects of the herbal compound may be mediated by the IL-17 signaling pathway, the TNF signaling pathway, and other signaling pathways involved in inflammatory response and immune regulation. Molecular docking showed that the three core target proteins produced stable binding to the two main active ingredients.ConclusionThe screening of effective herbal compounds for the clinical treatment of liver cancer based on NDCNN and genetic algorithms is a feasible approach and will provide ideas for the development of herbal medicines for the treatment of liver cancer and other cancers.