In order to explore the effect of nano Mg(OH)2(MH) on the flame retardancy and mechanical properties of nitrile-butadiene rubber (NBR) composite, molecular simulation was employed to examine the following properties of nano-MH/NBR composite, including thermal decomposition temperature, agglomeration and dispersion mechanism, shear behavior, stretcher strain and gyration radius of gyration. Discussion was performed on the microscopic mechanism of the flame retardancy and mechanical properties of nano MH/NBR composite. The simulation results indicated that the introduction of nano MH increased the thermal decomposition temperature of the composites by up to 28.92%, and that compared with the unmodified nano-MH/NBR composite, the thermal decomposition temperature of the conventionally-modified composite was increased by 19.22%. Besides, the crystal morphology, polarity and dispersion of nano-MH could be well optimized by hydrothermal modification. The Young's modulus of the composite was increased by 35.65% by adding nano-MH into pure NBR. The Young's modulus of the conventionally -modified nano-MH(101)/NBR and the hydrothermally-modified nano-MH(001)/NBR composite was 20.93 GPa and 21.43 GPa, respectively, an increase of 35.82% and 39.06% compared with unmodified nano-MH/NBR composite, respectively. To conclude, adding nano-MH could improve the flame retardancy and mechanical properties of NBR composite, and modified nano-MH performed better than unmodified nano-MH in NBR composite.