This study explores the synergies between advanced cooling technologies and photovoltaic systems, seeking to improve their overall efficiency and contribute to the broader goal of mitigating greenhouse gas emissions. To cool photovoltaic panels in more efficiently maner, understanding heat pipes, nanofluids, and panels interaction play key roles. For analysis and optimization, hybrid models of convolutional neural network (CNN) and firefly optimization algorithm are employed. The firefly optimization algorithm is used to optimize the thermosiphon heat pipe’s operational conditions, taking into account inputs such as the filling ratio, nanofluid concentration and panel angle. The study compared the predicted outcomes of a classic CNN model to laboratory experiments. While the CNN model was consistent with experimental findings, it struggled to predict high power values with precision. The proposed model improved high power value predictions by 4.05 W root mean square error (RMSE). The proposed model outperformed the classic CNN model for values greater than 50 W, with an RMSE of 3.95 W. The optimal values for the filling ratio, nanofluid concentration and panel angle were determined after optimization with the firefly algorithm. The research contributes to the advancement of renewable energy technologies and the optimization of photovoltaic panel cooling and energy production. Nanofluid with 1% mass concentration improves photovoltaic collector thermal efficiency due to its higher thermal conductivity coefficient. The photovoltaic collector’s electrical efficiency peaks in the morning, drops at noon due to temperature and radiation and recovers by morning. Electrical efficiency is best with nanofluid at 0.86%. Exergy efficiency closely matches electrical efficiency, with nanofluid at the optimal percentage achieving the highest efficiency and water cooling the lowest.