Diabetes, especially type 2, has been rapidly increasing all over the world. Although many drugs have been developed and used to treat diabetes, side effects and long-term efficacy are of great challenge. Therefore, natural health product and dietary supplements have been of increasing interest alternatively. In this regard, Chinese herbs and herbal products have been considered a rich resource of product development. Although increasing evidence has been produced from various scientific studies, the mechanisms of action are lacking. Here, we have proposed that many herbal monomers and formulae improve glucose homeostasis and diabetes through the BMID axis; B represents gut microbiota, M means mucosal immunity, I represents inflammation, and D represents diabetes. Chinese herbs have been traditionally used to treat diabetes, with minimal side and toxic effects. Here, we reviewed monomers such as berberine, ginsenoside, M. charantia extract, and curcumin and herbal formulae such as Gegen Qinlian Decoction, Danggui Liuhuang Decoction, and Huanglian Wendan Decoction. This review was intended to provide new perspectives and strategies for future diabetes research and product.