A whole-grain (WG) diet affects human health in multiple ways. However, the effect of WG on the gut microbiota of the elderly individuals is still largely unknown. In this study, WG did not affect the microbial α-diversity but had a profound impact on the microbes' abundance in the elderly individuals. WG increased the abundance of Verrucomicrobia and decreased the abundance of Firmicutes. The prediction of microbial function showed that glucose metabolism and lipid metabolism were inhibited. In addition, the effects of WG on the gut microbiota of normal-weight (NW) and overweight (OW) individuals were different. WG increased Verrucomicrobia in the NW group and decreased Firmicutes in the OW group. Meanwhile, the effect of WG on gut microbiota showed gender characteristics, Firmicutes/Bacteroidetes ratio was decreased in women, while Verrucomicrobia abundance was increased in men. The use of WG could improve the microbial composition and promote the growth of beneficial microbes, which may be beneficial to the health of the elderly individuals.