Cadmium (Cd) is highly toxic, even at very low concentrations, to both animals and plants. Pollen is extremely sensitive to heavy metal pollutants; however, less attention has been paid to the protection of this vital part under heavy metal stress. A pot experiment was designed to investigate the effect of foliar application of Se (1 mg/L) and Mo (0.3 mg/L) either alone or in combination on their absorption, translocation, and their impact on Cd uptake and its further distribution in Brassica napus, as well as the impact of these fertilizers on the pollen grains morphology, viability, and germination rate in B. napus under Cd stress. Foliar application of either Se or Mo could counteract Cd toxicity and increase the plant biomass, while combined application of Se and Mo solutions on B. napus has no significant promotional effect on plant root and stem, but reduces the seeds’ weight by 10–11%. Se and Mo have decreased the accumulated Cd in seeds by 6.8% and 9.7%, respectively. Microscopic studies, SEM, and pollen viability tests demonstrated that pollen grains could be negatively affected by Cd, thus disturbing the plant fertility. Se and Mo foliar application could reduce the toxic symptoms in pollen grains when the one or the other was sprayed alone on plants. In an in vitro pollen germination test, 500 μM Cd stress could strongly inhibit the pollen germination rate to less than 2.5%, however, when Se (10 μM) or Mo (1.0 μM) was added to the germination medium, the rate increased, reaching 66.2% and 39.4%, respectively. At the molecular level, Se and Mo could greatly affect the expression levels of some genes related to Cd uptake by roots (IRT1), Cd transport (HMA2 and HMA4), Cd sequestration in plant vacuoles (HMA3), and the final Cd distribution in plant tissue at the physiological level (PCS1).