In this work, an effective flame retardant consisting of nanoscale zinc oxide doped on the surface of hexagonal lamellar magnesium hydrate (ZO@MH) has been successfully synthesized via a hydrothermal process. Approximately 3-methacryloxypropyltrimethoxysilane (KH-570) is chosen as a modifier of ZO@MH for the purpose of enhancing the interfacial interaction between ZO@MH and the polypropylene (PP) matrix and reducing the agglomeration of ZO@MH. Afterwards, ZO@MH and KH-570 modified ZO@MH (KZO@MH) filled PP (PP/ZO@MH and PP/KZO@MH) composites are respectively prepared via the melt blending method. The flame retardant and smoke suppression properties of PP/ZO@MH and PP/KZO@MH composites are estimated by a cone calorimetry test (CCT). The peak value of the heat release rate of the PP/40KZO@MH composite is 327.0 kW/m2, which is 6.1% and 31.2% lower than that of the PP/40ZO@MH and PP/40MH composites, respectively. The lowest peak values of CO and CO2 production, 0.008 and 0.62 g/s, also appeared in the PP/40KZO@MH composite, which are 11.1% and 10.1% lower than those of the PP/40ZO@MH composite. Analysis of char residues indicates that nanoscale ZO and modification of KH-570 improve the amount and quality of char residues, which should be the main reason for the good flame retardant and smoke suppression properties of KZO@MH. Impact strength and nominal strain at break results show that the PP matrix is toughened by ZO@MH rather than KZO@MH. Tensile properties and the quantitative interfacial interaction calculated by the Turcsányi equation both prove the reinforcement of KZO@MH on the PP matrix.