The present study examines the turbulent flow of mixed convection heat transfer enhancement within a rectangular channel considering three different novel shapes of ribs (smooth, scalene, and curved‐side triangular). The investigations were conducted experimentally by developing a new test facility, while the numerical computations were carried out using the finite volume method. The experimental work involves constructing of the channel, ribs, and all equipment and measurement instruments. The numerical work is based on ANSYS FLUENT considering the k–ε turbulent model. The results are presented and compared in terms of Nusselt number, friction factor, and performance factors for Reynolds numbers ranging between 3000 and 12,000. By comparing the average values of the numerically obtained Nusselt number with experimental measurements, the data showed a close agreement with a maximum difference of 5%. It also found that scalene triangular ribs (STRs) provide better performance in terms of heat transfer, although introducing a slight increase in friction losses. STRs showed (20%) increase in Nusselt number compared with smooth channel, and 3%–6% increase in Nusselt number compared with curved‐side triangular ribs (CTRs). In contrast, CTRs have a lower friction factor value of 5% compared with STRs at a low value of a Reynolds number of 3000. Furthermore, the Nusselt number changes significantly (250% increase) by increasing the value of the Reynolds number from 3000 to 12,000. A thermal performance factor of up to 1.28 was achieved for the STRs at the lowest range of Reynolds' number of 3000. The findings from the present study are of practical importance for industries requiring heat transfer enhancement techniques to improve heat transfer equipment performance.