In multiple-input-multiple-output (MIMO) systems, the selection of receive and transmit antennas is not just effective in increasing system capacity, but also in reducing RF link costs and system complexity. The exhaustive algorithm, i.e. the joint transmit and receive antenna selection (JTRAS) with the best accuracy, can search all the subsets of both transmit and receive antennas in order to find the optimal solution. However, with the increase of the number of antennas, the computational complexity is too large and its applicability is limited. In this paper, the antennas are coded by fractional coding with the maximization of channel capacity as the basic criterion, and three intelligent algorithms, namely genetic algorithm, cat swarm algorithm and particle swarm algorithm, are applied for antenna selection. The simulation results demonstrate that all three algorithms can efficiently accomplish the antenna selection. In the end, we compare them in terms of speed, accuracy and complexity of the search in MIMO systems.