Antenna grouping algorithms are hybrids of transmit beamforming and spatial multiplexing. With antenna grouping, we can achieve combining gain through transmit beamforming, and high spectral efficiency through spatial multiplexing. In an independent identically distributed channel, the antenna grouping method has better bit error rate (BER) performance than the beamforming method. However, if the channel is correlated, then the BER performance of antenna grouping degrades. In that case, it is better to use beamforming instead of antenna grouping. In this article, we investigate the mode selection algorithms which select between the antenna grouping and the beamforming modes. By selecting a suitable mode for a given channel, we can achieve more robustness of the system performance. We introduce several mode selection criteria as well as a low complexity criterion which is derived from a low complexity antenna grouping algorithm. Simulation results show that the proposed mode selection algorithm performs better than the antenna grouping and the beamforming modes in various channel conditions.