BackgroundCastration‐resistant prostate cancer (CRPC) is refractory to hormone treatment and the therapeutic options are continuously advancing. This study aims to discover the anti‐CRPC effects and underlying mechanisms of small‐molecule compounds targeting topoisomerase (TOP) II and cellular components of DNA damage repair.MethodsCell proliferation was determined in CRPC PC‐3 and DU‐145 cells using anchorage‐dependent colony formation, sulforhodamine B assay and flow cytometric analysis of CFSE staining. Flow cytometric analyses of propidium iodide staining and JC‐1 staining were used to examine the population of cell‐cycle phases and mitochondrial membrane potential, respectively. Nuclear extraction was performed to detect the nuclear localization of cellular components in DNA repair pathways. Protein expressions were determined using Western blot analysis.ResultsA series of azathioxanthone‐based derivatives were synthesized and examined for bioactivities in which WC‐A13, WC‐A14, WC‐A15, and WC‐A16 displayed potent anti‐CRPC activities in both PC‐3 and DU‐145 cell models. These WC‐A compounds selectively downregulated both TOP IIα and TOP IIβ but not TOP I protein expression. WC‐A13, WC‐A14, and WC‐A15 were more potent than WC‐A16 on TOP II inhibition, mitochondrial dysfunction, and induction of caspase cascades indicating the key role of amine‐containing side chain of the compounds in determining anti‐CRPC activities. Furthermore, WC‐A compounds induced an increase of γH2AX and activated ATR‐Chk1 and ATM‐Chk2 signaling pathways. P21 protein expression was also upregulated by WC‐A compounds in which WC‐A16 showed the least activity. Notably, WC‐A compounds exhibited different regulation on Rad51, a major protein in homologous recombination of DNA in double‐stranded break repair. WC‐A13, WC‐A14, and WC‐A15 inhibited, whereas WC‐A16 induced, the nuclear translocation of Rad51.ConclusionThe data suggest that WC‐A compounds exhibit anti‐CRPC effects through the inhibition of TOP II activities, leading to mitochondrial stress‐involved caspase activation and apoptosis. Moreover, WC‐A13, WC‐A14, and WC‐A15 but not WC‐A16 display inhibitory activities of Rad51‐mediated DNA repair pathway which may increase apoptotic effect of CRPC cells.