We employ the direct numerical simulation to study the heat transfer behavior and flow structures in a vertical convection system with rough sidewalls. The parameters are chosen with Rayleigh number spanning the range of 1×108≤Ra≤3×1010 and Prandtl number fixed at 1.0. The results reveal that the impact of rough walls on the Nusselt number Nu and the Reynolds number Re is influenced by the height of the rough element h. When h is not sufficiently high, the roughness impedes the flows within the boundary layer and traps massive heat between rough elements, and both Nu and Re are lower than those in the smooth-wall case. However, the extent of the Nu and Re reduction regimes decreases as Ra increases. For sufficiently large Ra, the reduction regime for both Nu and Re may vanish, and roughness breaks up the limitation of the thermal boundary layer and facilitates the eruption of thermal plumes from roughness tips, resulting in the enhancement of both Nu and Re. Based on these results, the critical heights hc for Nu and hcr for Re are identified. Both exhibit similar scaling behavior with Ra, with hc consistently being larger than hcr for the same value of Ra.