Layered double hydroxide (LDH) was in situ modified by sorbic acid (SA) and used to reinforce hydrogenated nitrile rubber (HNBR) by direct blending method. SA connected LDH and HNBR via grafting/coordination mechanism. The results showed that SA could effectively improve the interaction between LDH and HNBR and significantly improve the dispersion of LDH in the rubber matrix. In addition, zinc sorbate (ZDS) formed by the reaction of SA and zinc oxide (ZnO) could self‐polymerize with the initiation of Dicumyl peroxide to form poly‐zinc sorbate, which is conducive to the reinforcement of rubber. The LDH/HNBR composite with SA loading of 20 phr has the best comprehensive mechanical properties, including tensile strength of 39.31 MPa, tear strength of 62.38 kN/m, shore A hardness of 85.1°, and elastic modulus of 20.14 MPa, which is significantly better than the LDH/HNBR composite without addition of SA. Our work provides a practicable and effective methodology for in situ modification of LDH and fabrication of high‐performance rubber composites.