Development of high‐performance thermosetting resins by adding environmentally friendly flame retardant to heat‐resistant resins without deteriorating their outstanding thermal stability is an important research direction. Here, a unique hybrid (GHNT) consisting of graphene oxide (GO) and halloysite nanotubes (HNT) was synthesized, and then a series of composites based on cyanate ester (CE) resin were fabricated. The effects of GHNT on the heat resistance, flame retardancy, and smoke suppression of GHNT/CE composites were intensively investigated. The GHNT/CE composite with 5.0 wt % GHNT not only has about 15.1 °C higher initial degradation temperature, but also shows 54.6% or 37.9% lower peak heat release rate or maximum smoke density than CE resin. These results clearly demonstrate that GHNT is not the simple combination of GO and HNT; instead, it obviously shows positive synergistic effects in simultaneously improving the flame retardancy and thermal resistance of CE resin. The improved flame retardancy could be attributed to condensed‐phase mechanisms, including increasing char yield, building a dense char layer, and free radical scavenging. © 2018 Wiley Periodicals, Inc. J. Appl. Polym. Sci. 2018, 135, 46587.