Hyperbranched polyborate (HBPB) is a novel polymer with highly branched structures, plenty of functional end groups, and excellent thermal resistance, which could be used as an effective modifier for thermoset resin to greatly improve the thermal resistance and toughness of the resin, simultaneously. With the assistance of benzoxazine (BOZ) resin which has low viscosity, high thermal resistance, and no by‐product release upon curing, modification effects of HBPB on the thermal resistance and toughness of phenolic resin (PR) can be adequately realized. The optimal combination of HBPB and BOZ modified PR (PR‐BOZ‐HBPB) is obtained by the orthogonal test. The PR‐BOZ‐HBPB blend prepared with the optimal combination (90 portion of PR, 10 portion of BOZ, and 5 portion of HBPB) has a char yield up to 67.4%; meanwhile, flexural strength, interlaminar shear strength, flexural modulus of this glass/PR‐BOZ‐HBPB composite are higher than those of glass/PR composite by 17.6%, 23.3%, and 9.6%, respectively. The novel thermoset resin, PR‐BOZ‐HBPB blend, which consists of good processability, great thermal and mechanical properties, is obtained. POLYM. COMPOS., 33:1960–1968, 2012. © 2012 Society of Plastics Engineers