The stability of the composite structures formed by coal pillar and roof rock is of great significance to safe production and sustainable development of coal mines. In order to explore the failure and instability mechanisms of coal–rock combinations (CRCs) with varying rock-to-coal height ratios, uniaxial compression tests of CRCs with varying rock-to-coal height ratios were performed via laboratory tests and numerical simulation of particle flow tests, and the mechanical response and failure mechanisms of CRC were comprehensively investigated with regard to their strength, failure characteristics, crack and energy evolution. The results show that the stress thresholds for the crack initiation, uniaxial compressive strength, and elastic modulus of CRCs rose with the decreasing coal-to-rock height ratio, and the fragmentation degree of the coal samples increased with rock-to-coal height ratio. The instability and failure of CRCs are the result of the interaction between the strength of sandstone and coal at the interface and the strength of sandstone and coal far from the interface region; in addition, they are influenced by the distribution range of microfractures during the loading process. The point effect and slip effect formed by coal failure cause sandstone split failure and shear failure, respectively. The number of cracks, macrocrack length, total input energy, elastic strain energy, and dissipated strain energy all reduce first and then increase as the coal thickness reduces. CRCs still have a certain load-bearing capacity in the post-peak stage, mainly due to their strong load-bearing skeleton structure and the friction between particles in the fracturing area, which can resist external forces.