Understanding the role of aggregates in the compressive strength of brittle composites is crucial for optimizing construction material usage. In this study, we employed the rigid-body-spring-network model to determine the compressive strengths of both coarse aggregate and mortar in concrete. Through uniaxial compression loading tests on concrete, considering Young’s modulus and compressive strength of the coarse aggregate as variables, we elucidated the mechanism by which the coarse aggregate’s physical properties influence concrete's compressive strength, illustrated using a straightforward diagram. When the coarse aggregate’s Young's modulus is lower than that of the mortar, the stress transfer path within the mortar bends and cracks more rapidly, significantly lowering strength. Conversely, if the coarse aggregate’s Young’s modulus exceeds that of the mortar, stress becomes concentrated in the aggregate. Considering these aspects, the compressive strength of concrete can be optimized by changing the properties of the mortar according to the properties of the aggregates.