SummaryIn this study, new cooperative‐quadrature spatial modulation techniques based on Euclidean distance and capacity optimized antenna selection techniques (EDAS‐CQSM and COAS‐CQSM) are proposed for cooperative communication systems. The considered cooperative system consists of three terminals that are the source terminal (ST), relay terminal (RT), and destination terminal (DT). The decode‐and‐forward (DF) cooperative technique is considered to retransmit the signal at the RT. In order to improve the performance of the conventional QSM demodulator in the RT and DT, both antenna selection techniques have been used in both the ST and RT to reduce performance degradation due to error propagation of the DF scheme. Therefore, the overall performance of the considered CQSM system has been increased. Furthermore, computer simulations have shown that the proposed EDAS/COAS‐CQSM systems have better the performances than the cooperative spatial modulation (CSM), CQSM, EDAS‐CSM, COAS‐CSM, and traditional DF‐based cooperative system with the same bit rate.