In this work, we synthesized a novel organic-inorganic hybrid, DOPO-AS@M(OH)(OCH3) (DOPO-AS@M (M = Co,Ni)), and utilized it to prepare the epoxy resin (EP) composite with enhanced flame retardancy and mechanical properties. Specifically, the DOPO-AS was synthesized using 3-aminopropyltriethoxysilane (APTES), salicylaldehyde (Salal), and 9,10-dihydro-9-oxa-10-phosphaphenanthrene-10-oxide (DOPO). Subsequently, the DOPO-AS was utilized as a surface modification of the novel 2D nanosheet M(OH)(OCH3) and the obtained organic-inorganic hybrid was used to prepare the EP composites. The results showed that the addition of only 5 wt% DOPO-AS@M enabled EP to achieve the UL-94 of V-1 rating, with the limiting oxygen index (LOI) increasing from 28.7–31.0%. The peak heat release rate (pHRR) of EP/DOPO-AS@M 5% was 1007.5 kW/m2, with the reduction of 48% compared to pure EP. In addition, the elongation at break of EP/DOPO-AS@M 5% increased from 3.9–4.2%. The DOPO-AS@M nanohybrid proposed in this study providing the new solution for simultaneously enhanced the flame retardancy and mechanical properties of EP, thus broadening the application of EP in the market.