We consider forced convective heat transfer in a channel flow with a rough boundary. The roughness of the domain is expressed as the product of length scale ε and arbitrary smooth function h. The boundaries of the channel are assumed as hot (top) and cold (bottom) walls. We derive an effective model reflecting the rough boundary effects, and analyze the average Nusselt number and wall laws (effective boundary conditions). Consequently, we provide the criterion for the shape of the rough boundary to enhance the heat transfer. We further confirm that the Robin–type effective boundary condition is sufficient to describe the effect of the rough layer on the heat transfer.