How to reduce the negative transport externalities, especially its carbon emissions, without having significant negative influence on economic and social development is the key for sustainable development in China. This paper explores the impacts of China’s recent modal shift policy on carbon emissions, summaries experience from China, and points out future development directions. The paper first compares the different energy consumption and carbon emissions between the road freight transport and the railways in China, and then has a scenarios analysis on China’s energy consumption and carbon emissions of the transport sector in 2025. The latest progress and major problems of modal shift policy in China are presented, and a methodology to address this problem is also proposed. Based on the methodology, we compare the benefits and costs brought by modal shift policy in the case of Ordos, Inner Mongolia. Based on the results, principles and suggestions on how to design and implement more efficient modal shift policy are proposed. We find that road transport is the most polluting mode among various modes of transport, and the railway transport has the least carbon emissions. Furthermore, the modal shift policy plays a positive role in carbon emissions, but the costs caused by the policy are higher than the benefits at some circumstances. Moreover, to achieve the sustainable modal shift policy by relying on the feasible market mechanism, together with scientific and effective regulation, instead of “one size for all” administrative policy, are likely the way forward.