A continuum-field–multi-phase-field method was adopted to simulate the microstructure evolution of a Nd–Fe–B magnet during the direct-chill casting process, and the influence of temperature gradient on the microstructure evolution was focused on. Contrary to the case of uniform temperature field, the microstructure and solute atoms showed an obvious non-uniform distribution under the temperature gradient. Three totally different zones, i.e., the amorphous phase along with some nanocrystalline grains, the uniform fine grains, and the coarse grains, were discovered. Besides this, the effects of the cooling rate and the external magnetic field on the microstructure evolution were investigated. The higher the cooling rate was, the smaller the overall grain size was, and the higher the uniformity of the microstructure. The external magnetic field had a slight grain refinement effect for Nd2Fe14B (T1) phase and could restrain the formation of an amorphous phase and a Nd1+εFe4B4 (T2) phase to some extent.