Aminopeptidase N (APN, CD13) is closely associated with the development and progression of cancer. Previous studies suggested APN as a biomarker for cancer stem cells. APN inhibitors have been intensively evaluated as chemosensitizers for cancer treatments. In the present study, tetrahydro-β-carboline scaffold was introduced to the structure of APN inhibitors. The synthesized compounds showed potent enzyme inhibitory activities compared with Bestatin, an approved APN inhibitor, in cell-based enzymatic assay. In combination with chemotherapeutic drugs, representative APN inhibitor molecules D12, D14 and D16 significantly improved the antiproliferative potency of anticancer drugs in the in vitro tests. Further mechanistic studies revealed that the anticancer effects of these drug combinations are correlated with decreased APN expression, increased ROS level, and induction of cell apoptosis. The spheroid-formation assay and colony-formation assay results showed effectiveness of Paclitaxel-APN inhibitor combination against breast cancer stem cell growth. The combined drug treatment led to reduced mRNA expression of OCT-4, SOX-2 and Nanog in the cancer stem cells tested, suggesting the reduced stemness of the cells. In the in vivo study, the selected APN inhibitors, especially D12, exhibited improved anticancer activity in combination with Paclitaxel compared with Bestatin. Collectively, potent APN inhibitors were discovered, which could be used as lead compounds for tumor chemo-sensitization and cancer stem cell-based therapies.