Thermoelectrics, which convert waste heat to electricity, offer an attractive pathway for addressing an important niche in the globally growing landscape of energy demand. Research to date has focused on reducing the thermal conductivity relative to the bulk. Si nanowires (NWs) have received exceptional attention due to their low-dimensionality, abundance of availability, and high carrier mobility. From thermal transport point of view, the thermal conductivity of Si NWs strongly depends on the detailed surface structure, such as roughness and surface orientation. Here, direct molecular dynamics simulations and theoretical models are used to investigate the thermal transport in Si NWs with diverse surface orientations. Our results show that the thermal conductivity of Si NWs with different surface orientation can differ by as large as 2.7~4.2 times, which suggests a new route to boost the thermoelectric performance. Using the full spectrum theory, we find that the surface orientation, which alters the distribution of atoms on the surface and determines the degree of phonon coupling between the core and the surface, is the dominant mechanism. Furthermore, using spectral thermal conductivity, the remarkable difference in the thermal conductivity for different surface orientation is found to only stem from the phonons in the medium frequency range, with minor contribution from low and high frequency phonons.