Group IV alloys have been long viewed as homogeneous random solid solutions since they were first perceived as Si-compatible, direct-band-gap semiconductors 30 years ago. Such a perception underlies the understanding, interpretation and prediction of alloys' properties. However, as the race to create scalable and tunable device materials enters a composition domain far beyond alloys' equilibrium solubility, a fundamental question emerges as to how random these alloys truly are. Here we show, by combining statistical sampling and large-scale ab initio calculations, that GeSn alloy, a promising group IV alloy for mid-infrared technology, exhibits a clear, short-range order for solute atoms within its entire composition range. Such short-range order is further found to substantially affect the electronic properties of GeSn. We demonstrate the proper inclusion of this short-range order through canonical sampling can lead to a significant improvement over previous predictions on alloy's band gaps, by showing an excellent agreement with experiments within the entire studied composition range. Our finding thus not only calls for an important revision of current structural model for group IV alloy, but also suggests short-range order may generically exist in different types of alloys.