Recently, Wi-Fi-based human activity recognition using channel state information (CSI) signals has gained popularity due to its potential features, such as passive sensing and adequate privacy. The movement of various body parts in between Wi-Fi signals' propagation path generates changes in the signal reflections and refraction, which is evident from the CSI variations. In this paper, we analyzed the relationship between human activities and properties (amplitude and phase) of Wi-Fi CSI signals on multiple receiving antennas and discovered the signal properties which vary remarkably in response to human movement. The variation of the received signal among multiple antennas shows different sensitivity to human activities, directly impacting the recognition performance. Therefore, to recognize human activities with better efficiency, we proposed an adaptive antenna elimination algorithm that automatically eliminates the non-sensitive antenna and keeps the sensitive antennas following different human activities. Furthermore, the correlation of the statistical features extracted from the amplitude and phase of the selected antennas' CSI signal was analyzed, and a sequential forward selection was utilized to find the best subset of features. Using such a subset, three machine learning algorithms were employed on two available online datasets to classify various human activities. The experimental results revealed that even when using easy-toimplement, non-deep machine learning, such as random forest, the recognition system based on the proposed adaptive antenna elimination algorithm achieved a superior classification accuracy of 99.84% (line of sight) on the StanWiFi dataset and 97.65% (line-of-sight) / 93.33% (non-line-of-sight) on another widely applied multi-environmental dataset at a fraction of the time cost, demonstrating the robustness of the proposed algorithm.