BackgroundXanthone derivatives have a wide range of pharmacological activities, such as those involving antibacterial, antiviral, antimalarial, anthelmintic, anti-inflammatory, antiprotozoal, and anticancer properties. Among these, we investigated the anticancer properties of xanthone. This research aimed to analyze the biological activity of ten novel xanthone derivatives, to investigate the most contributing-descriptors for their cytotoxic activities, and to examine the possible mechanism of actions of xanthone compound through molecular docking.Materials and methodsThe cytotoxic tests were carried out on WiDR and Vero cell lines, by a 3-(4,5-dimethylthiazole-2-yl)-2,5-diphenyl-tetrazolium bromide (MTT) assay method. The structural features required for xanthone’s anticancer activity were conducted by using the semi-empirical Austin Model-1 method, and continued with quantitative structure-activity relationship (QSAR) analysis using BuildQSAR program. The study of the possible mechanism of actions of the selected xanthone compound was done through molecular docking with PLANTS.ResultsThe three novel xanthone derivatives (compounds 5, 7, and 8) exhibited cytotoxic activity with compound 5 showed the highest degree of cytotoxicity at concentration 9.23 µg/mL (37.8 µM). The following best equation model was obtained from the BuildQSAR calculation: log 1/IC50 = −8.124 qC1 −35.088 qC2 −6.008 qC3 + 1.831 u + 0.540 logP −9.115 (n = 10, r = 0.976, s = 0.144, F = 15.920, Q2 = 0.651, SPRESS = 0.390). This equation model generated 15 proposed new xanthone compounds with better-predicted anticancer activities. A molecular docking study of compound 5 showed that xanthone formed binding interactions with some receptors involved in cancer pathology, including telomerase, tumor-promoting inflammation (COX-2), and cyclin-dependent kinase-2 (CDK2) inhibitor.ConclusionThe results suggested that compound 5 showed the best cytotoxic activity among the xanthone derivatives tested. QSAR analysis showed that the descriptors contributed to xanthone’s cytotoxic activity were the net atomic charge at qC1, qC2, and qC3 positions, also dipole moment and logP. Compound 5 was suspected to be cytotoxic by its inhibition of telomerase, COX-2, and CDK2 receptors.