Supplementation with benzoic acid (BA) in animal feed can reduce feeds’ acid-binding capacity, inhibit pathogenic bacterial growth, enhance nutrient digestion, and increase intestinal enzyme activities. This study aimed to investigate the effects of different doses of BA on the growth performance, rumen fermentation, and rumen microbiota of weaned Holstein dairy calves. Thirty-two Holstein calves at 60 days of age were randomly assigned into four groups (n = 8): a control group (fed with a basal diet without BA supplementation; CON group) and groups that were supplemented with 0.25% (LBA group), 0.50% (MBA group), and 0.75% (HBA group) BA to the basal diet (dry matter basis), respectively. The experiment lasted for 42 days, starting at 60 days of age and ending at 102 days of age, with weaning occurring at 67 days of age. Supplementation with BA linearly increased the average daily gain of the weaned dairy calves, which was significantly higher in the LBA, MBA, and HBA groups than that in the CON group. The average daily feed intake was quadratically increased with increasing BA supplementation, peaking in the MBA group. Supplementation with BA linearly decreased the feed-to-gain (F/G) ratio, but did not affect rumen fermentation parameters, except for the molar proportion of butyrate and iso-butyrate, which were linearly increased with the dose of BA supplementation. Compared with the CON group, the molar proportions of iso-butyrate in the LBA, MBA, and HBA groups and that of butyrate in the HBA group were significantly higher than those in the CON group. Supplementation with BA had no significant effect on the alpha and beta diversity of the rumen microbiota, but significantly increased the relative abundances of beneficial bacteria, such as Bifidobacterium, and reduced those of the harmful bacteria, such as unclassified_o__Gastranaerophilales and Oscillospiraceae_UCG-002, in the rumen. Functional prediction analysis using the MetaCyc database revealed significant variations in the pathways associated with glycolysis across groups, including the GLYCOLYSIS-TCA-GLYOX-BYPASS, GLYCOL-GLYOXDEG-PWY, and P105-PWY pathways. In conclusion, BA supplementation improved the composition and function of rumen microbiota, elevated the production of butyrate and iso-butyrate, and increased the growth performance of weaned Holstein dairy calves.