The aim of this work was to study the effect of carboxyl‐terminated acrylonitrile butadiene (CTBN) liquid rubber and alumina nanoparticles (Al2O3) on the fracture toughness of an epoxy resin. Hybrid nanocomposite samples were prepared by the inclusion of the optimum concentration of CTBN (15 phr) and 0–10 phr Al2O3 into the epoxy matrix. Samples were examined using Fourier transform infrared spectroscopy, scanning electron microscopy, and fracture toughness test. Fractal dimension values were also estimated from the edge of the fracture surfaces for the hybrid samples. The maximum value of critical stress intensity factor,
K
IC, was found for the non‐hybrid sample containing 15 phr CTBN and the
G
IC values showed an increasing trend as it was expected from the increase in
K
IC values and decrease in Young's modulus. The maximum
K
IC for the hybrid samples was obtained for the one containing 7 phr Al2O3 and an almost similar trend was observed for
G
IC, accompanied with the significant changes in the fracture surface morphology. These were consistent with the fractal dimension results, where the sample containing 7 phr Al2O3 exhibited the maximum roughness. Therefore, the hybrid system of epoxy/CTBN (15 phr)/Al2O3 (7 phr) was found to be the optimum formulation. POLYM. COMPOS., 40:2700–2711, 2019. © 2018 Society of Plastics Engineers