Novel glycidyl methacrylate–butyl acrylate–maleic anhydride (GBM) terpolymers with different molecular weights were synthesized by radical polymerization and characterized using fourier transform infrared, nuclear magnetic resonance (1H‐NMR and 13 C‐NMR), and gel permeation chromatography. Each GBM terpolymer was used to modify aluminum nitride (AlN), and the modified AlN, coded as AlN(GBM), was added to 2,2′‐bis(4‐cyanatophenyl)isopropylidene (CE) resin for preparing composites. Composites based on original AlN or γ‐(2,3‐epoxypropoxy)propyltrimethoxysilane‐modified AlN (AlN(K)) were also prepared for comparison. Although GBM and γ‐(2,3‐epoxypropoxy)propyltrimethoxysilane have similar reactive groups, the results indicate that GBM shows more attractive integrated advantages, reflected by the fact that CE/AlN(GBM) composites have better thermal stability, higher thermal conductivity, and higher glass transition temperature than those of CE/AlN(K). These properties result from better dispersion of fillers, improved interfacial adhesion between fillers and CE resin, and increased cross‐linking density. This study demonstrates that the nature of the coupling agents is an important factor to develop high performance composites for cutting‐edge industries. Copyright © 2011 John Wiley & Sons, Ltd.