Multiwalled carbon nanotubes (MWCNTs) were modified by oxidizing agent to obtain O-MWCNTs, and the surface of it was coated with active group. 4,4′-Diaminodiphenylmethane bismaleimide (MBMI) was used as matrix, 3,3′-diallyl bisphenol A (BBA) and bisphenol-A diallyl ether (BBE) were used as reactive diluent, polyethersulfone (PES) as toughening agent, and O-MWCNTs as modifier; OMWCNT/PES-MBAE composite was prepared through in situ sol-gel method. The effect of PES and OMWCNTs on the mechanical and dielectric properties of composite was analyzed, and the microstructure was examined by transmission electron microscope (TEM) and scanning electron microscope (SEM). The mechanism of composite toughened by PES and OMWCNTs was observed and analyzed. The results showed that Diels-Alder reaction between MBMI and allyl compounds occurred completely and unsaturated double bond disappeared. O-MWCNTs and PES resin dispersed smoothly in polymer matrix and were used as reinforcement, and PES resin and O-MWCNTs could synergistically improve the properties of the composite and exhibited a typical ductile fracture. The impact and bending strengths were 16.09 kJ/m2 and 153.57 MPa, which were 74.32% and 53.08% higher than those of the MBAE matrix, respectively, and the dielectric constant and the dielectric loss were 3.76 (100 Hz) and 2.79 × 10−3 (100 Hz), when the content of PES was 2 wt% and O-MWCNTs was 0.02 wt%. The outstanding properties of the material made it play an important role in high-performance insulating material applications.