Background Nasopharyngeal carcinoma (NPC) is among the most common head and neck malignancies seen among adults in Malaysia. Therefore, discovery of novel anti-cancer herbal drugs is of importance. In this study, the cytotoxic effect was conducted on a traditional Chinese herbal prescription (Xiao Xian Xiong Decoction (XXXD) that is made up of 3 Chinese herbal medicines, namely Huanglian (Coptidis Rhizome), Banxia (Pinellia Rhizome), Gualuo (Fructus Trichosanthis).Methods The cytotoxic effect of the individual herb and in combination of two and three herbs was studied on 8 nasopharyngeal cancer cell lines. Global gene expression analysis was carried on extracted RNA using nCounter XT Gene Expression Assay.Results TWO-1, TWO-4, HONE-1, SUNE-1, CNE-2, HK-1, CNE-1 and C666-1 treated with Huanglian, the IC50 values obtained were 24.48, 11.77, 4.48, 10.72 6.32, 11.10, 6.77 and 27.30 µg/ml, respectively. For combination of Huanglian and Banxia, the IC50 values obtained were 74.09 µg/ml (TWO-1), 25.80 µg/ml (TWO-4), 38.10 µg/ml (HONE-1), 29.46 µg/ml (SUNE-1), 19.0 µg/ml (CNE-2) and 20.12 µg/ml (HK-1) but did not exert 50% cell killing in CNE-1 and C666-1 cell lines. The IC50 value attained for the combination of Huanglian and Gualuo was 40.70 µg/ml in HONE-1 cell line. The IC50 values obtained for XXXD (triple combination of Huanglian, Banxia and Gualuo)-treated in HONE-1 and CNE-2 cell lines were 88.55 and 92.42 µg/ml, respectively. Out of all these 7 groups of herbal samples, Huanglian showed the highest cytotoxicity against 8 NPC cell lines with the lowest IC50 value of 4.48 µg/ml recorded in HONE-1. Global gene expression showed Huanglian significantly downregulated genes associated with cell cycle arrest and apoptosis, and thus inhibit HONE-1 cell growth.Conclusions This study suggest that Huanglian could be a potent anticancer herb targeting HONE-1 cancer cell line.