Sensor-based human activity recognition (HAR) has attracted interest both in academic and applied fields, and can be utilized in health-related areas, fitness, sports training, etc. With a view to improving the performance of sensor-based HAR and optimizing the generalizability and diversity of the base classifier of the ensemble system, a novel HAR approach (pairwise diversity measure and glowworm swarm optimization-based selective ensemble learning, DMGSOSEN) that utilizes ensemble learning with differentiated extreme learning machines (ELMs) is proposed in this paper. Firstly, the bootstrap sampling method is utilized to independently train multiple base ELMs which make up the initial base classifier pool. Secondly, the initial pool is pre-pruned by calculating the pairwise diversity measure of each base ELM, which can eliminate similar base ELMs and enhance the performance of HAR system by balancing diversity and accuracy. Then, glowworm swarm optimization (GSO) is utilized to search for the optimal sub-ensemble from the base ELMs after pre-pruning. Finally, majority voting is utilized to combine the results of the selected base ELMs. For the evaluation of our proposed method, we collected a dataset from different locations on the body, including chest, waist, left wrist, left ankle and right arm. The experimental results show that, compared with traditional ensemble algorithms such as Bagging, Adaboost, and other state-of-the-art pruning algorithms, the proposed approach is able to achieve better performance (96.7% accuracy and F1 from wrist) with fewer base classifiers.