The objective of this study was to investigate the effect of waste milk with antibiotic residue on rumen fermentation and rumen bacterial composition of dairy calves during pre-weaned and post-weaned periods. A total of 24 Holstein male calves (43.4 ± 0.93 kg body weight, mean ± standard error) were allocated into four blocks based on birth date. Dairy calves were supplied 100% milk replacer (MR, n = 8), 50% milk replacer mixed with 50% waste milk (MM, n = 8), or 100% waste milk (WM, n = 8). Ruminal samples were collected at 49 and 63 days of age and then subjected to determinations of pH value, volatile fatty acids (VFA), ammonia nitrogen (NH3–N) and 16S rRNA gene amplicon sequencing. The results showed that feeding WM had no effect on the pH value, the concentrations of VFA (acetic acid, propionic acid, butyric acid, isovaleric acid, valeric acid), and NH3–N in dairy calves compared to feeding MR. However, from 49 to 63 days of age, the pH value (p < 0.001) was significantly increased, while the levels of total VFA (p = 0.004), acetic acid (p = 0.01), propionic acid (p = 0.003) and valeric acid (p < 0.001) were significantly decreased. For rumen microorganisms, there was no differences in bacterial diversity among the treatments. But the relative abundance of Veillonellaceae was significantly lower (p = 0.05) in the calves fed WM than that from MR group at 49 days of age; however, no difference was detected at 63 days of age. Feeding WM to calves tended to reduce family Veillonellaceae and genus Olsenella in the rumen at 49 days of age (p = 0.049). Analysis of temporal changes in rumen bacteria based on alpha-diversity and beta-diversity as well as the microbial relative abundances did not exhibit any difference. In addition, relative abundances of Clostridia_UCG-014, Prevotella, Syntrophococcus, Eubacterium_nodatum_group, Pseudoramibacter and Solobacterium were correlated with rumen pH value and the concentrations of TVFA, propionic acid, isovaleric acid, valeric acid and NH3–N. In conclusion, compare to MR, calves supplied with WM had little changes on the rumen pH value, NH3–N or VFAs contents. Additionally, limited effects could be found on rumen microbiota in the calves fed WM. However, further studies needed to explore if there exist any long-term effects of early-life rumen microbiota modulation on dairy cows.