The purpose of this study was to investigate the effects of replacing a portion of whole-plant corn silage with straw on the rumen microbial community structure and carbohydrate-active enzyme activity. The experiment employed a single-factor randomized trial design, with eight late-lactation Chinese Holstein dairy cows being randomly divided into two groups of four replicates each. The control group (CS group) was fed a diet consisting of alfalfa silage and a mixture of alfalfa and whole-plant corn silage, while the experimental group (RS group) received a diet in which one-third of the corn silage was replaced with straw while keeping the other components unchanged. The experiment lasted for a total of 21 days, with a pre-feeding period of 14 days and a formal period of 7 days. The rumen fluid collected on day 21 was used for the rumen fermentation parameters and metagenomic analysis. The concentrations of acetic acid, propionic acid, butyric acid, and total volatile fatty acids (TVFA) in the rumen of RS group cows were significantly lower than those in the CS group (p < 0.01). The ratio of acetic acid to propionic acid was significantly higher in the RS group compared to the CS group (p < 0.01). Metagenomic sequencing revealed that at the genus level, compared to the CS group, the abundance of unclassified bacteria, Bacteroides, Alistipes, Butyrivibrio, Chlamydia, Fibrobacter, unclassified Ruminococcaceae, and unclassified Bacteroidetes in the rumen of RS group cows increased, while the abundance of Eubacterium decreased ([LDA > 3.6], p < 0.05). Compared to the CS group, the enzymatic activities of α-L-arabinofuranosidase (EC3.2.1.55), β-xylosidase (EC3.2.1.37), β-glucosidase (EC3.2.1.21), β-glucosylceramidase (EC3.2.1.45), xylanase (EC3.2.1.8), and arabinanase (EC3.2.1.99) in the rumen of RS group cows increased (p < 0.05). According to the correlation analysis, Alistipes, Bacteroides, and Butyrivibrio showed a significant negative correlation with propionic acid (p < 0.05) and a significant positive correlation with the acetic acid-to-propionic acid ratio (p < 0.05). They also showed a significant positive correlation with GH2, GH3, GH20, GH29, GH43, GH78, GH92, CE1, GT4, β-glucosidase (EC3.2.1.21), α-L-arabinofuranosidase (EC 3.2.1.55), β-xylosidase (EC 3.2.1.37), β-glucosylceramidase (EC 3.2.1.45), xylanase (EC 3.2.1.8), and arabinanase (EC 3.2.1.99) (p < 0.05). In summary, straw can not only alter the composition and structure of the rumen microbiota in cows but also affect the relative abundance of CAZymes at different levels within the rumen. Cows may, thus, potentially improve the degradation efficiency of straw diets by increasing the abundance of certain rumen microbiota and enzymes.