2′-Z auraptene (1) is a synthesized monoterpene coumarin with anticancer activity against human gastric cancer cells. In order to find new potential anticancer agent, Mucor polymorphosporus was used to transform cis-auraptene. Four new terpene coumarins with notable changes in the skeletal backbone, 2′-Z auraptene A-D (2–5), were obtained and evaluated for their antiproliferative effects against human normal gastric epithelium cells and human gastric cancer cells. These new compounds showed selective cytotoxic activity against MGC-803 cells with IC50 values from 0.78 ± 0.13 to 10.78 ± 1.83 μM and the therapeutic index could also be significantly improved (TI = 59.0) compared with that of 1 (TI = 5.5). The structures of these metabolites were elucidated through extensive spectroscopic methods, and the possible biotransformation pathway of 1 by Mucor polymorphosporus was also proposed. Furthermore, the mechanism of the antiproliferative effects against MGC-803 cells of the most potent compound, 2′-Z auraptene A (2), was characterized. Annexin V/PI staining and abnormal expression of apoptosis-related protein suggested that compound 2 induces apoptosis in gastric cancer MGC-803 cells. Therefore, it is possible that compound 2 has the potential to be applied in gastric cancer therapy.Electronic supplementary materialThe online version of this article (10.1007/s12272-018-1028-0) contains supplementary material, which is available to authorized users.