This study was conducted to develop a new composite heterogeneous catalyst, anthill‐eggshell‐Ni‐Co mixed oxides (AENiCo), for biodiesel production via transesterification of waste frying oil. The catalyst was synthesized using a co‐precipitation method and was characterized by Brunauer‐Emmett‐Teller (BET) surface area analysis, Fourier transform infrared (FTIR) spectroscopy, scanning electron microscopy (SEM), X‐ray diffraction (XRD) and X‐ray fluorescence (XRF). The effects of parameters affecting the catalytic reaction, reaction temperature (40–80°C), reaction time (1–5 h), catalyst loading (1–9 wt%), and methanol‐to‐oil ratio (3:1–15:1) were investigated. The stability of the spent catalyst was also investigated during reuse. The data obtained showed that the maximum biodiesel yield of 90.23% could be obtained at a reaction temperature of 70°C, reaction time of 2 h, with a catalyst loading of 3 wt% and methanol‐to‐oil ratio of 12:1. The stability study showed that the AENiCo catalyst could be reused for up to four cycles. © 2018 Society of Chemical Industry and John Wiley & Sons, Ltd