The root rot disease causes a great economic loss, and the disease severity usually increases as ginseng ages. However, it is still unclear whether the disease severity is related to changes in microorganisms during the entire growing stage of American ginseng. The present study examined the microbial community in the rhizosphere and the chemical properties of the soil in 1–4-year-old ginseng plants grown in different seasons at two different sites. Additionally, the study investigated ginseng plants' root rot disease index (DI). The results showed that the DI of ginseng increased 2.2 times in one sampling site and 4.7 times in another during the 4 years. With respect to the microbial community, the bacterial diversity increased with the seasons in the first, third, and fourth years but remained steady in the second year. The seasonal changing of relative abundances of bacteria and fungi showed the same trend in the first, third, and fourth years but not in the second year. Linear models revealed that the relative abundances of Blastococcus, Symbiobacterium, Goffeauzyma, Entoloma, Staphylotrichum, Gymnomyces, Hirsutella, Penicillium and Suillus spp. were negatively correlated with DI, while the relative abundance of Pandoraea, Rhizomicrobium, Hebeloma, Elaphomyces, Pseudeurotium, Fusarium, Geomyces, Polyscytalum, Remersonia, Rhizopus, Acremonium, Paraphaeosphaeria, Mortierella, and Metarhizium spp. were positively correlated with DI (P < 0.05). The Mantel test showed that soil chemical properties, including available nitrogen, phosphorus, potassium, calcium, magnesium, organic matter, and pH, were significantly correlated to microbial composition. The contents of available potassium and nitrogen were positively correlated with DI, while pH and organic matter were negatively correlated with DI. In summary, we can deduce that the second year is the key period for the shift of the American ginseng rhizosphere microbial community. Disease aggravation after the third year is related to the deterioration of the rhizosphere microecosystem.