In the present work, a highly efficient incompressible flow solver with a semi-implicit time advancement on a fully staggered grid using a high-order compact difference scheme is developed firstly in the framework of approximate factorization. The fourth-order compact difference scheme is adopted for approximations of derivatives and interpolations in the incompressible Navier-Stokes equations. The pressure Poisson equation is efficiently solved by the fast Fourier transform (FFT). The framework of approximate factorization significantly simplifies the implementation of the semi-implicit time advancing with a high-order compact scheme. Benchmark tests demonstrate the high accuracy of the proposed numerical method. Secondly, by applying the proposed numerical method, we compute turbulent channel flows at low and moderate Reynolds numbers by direct numerical simulation (DNS) and large eddy simulation (LES). It is found that the predictions of turbulence statistics and especially energy spectra can be obviously improved by adopting the high-order scheme rather than the traditional second-order central difference scheme.