A novel artificial bee colony (ABC) algorithm to detect structural damage via modal and frequency analyses is proposed (named as TCABC algorithm). Compared to the standard ABC algorithm, tabu search method and chaotic search method are adopted in the proposed algorithm to enhance the exploration and exploitation ability. The tabu search method uses a memory function to avoid the solution being trapped in a local minimum, which increases the exploitation ability. Chaotic search method generates more searching points for finding the global minimum, which increases the exploration ability. Additionally, the first roulette wheel selection is replaced by the tournament selection to enhance the global searching ability of the TCABC algorithm. Several explicit test functions and an implicit damage detection function are employed to check the numerical results obtained from ABC and TCABC algorithms. Afterward, the damage detection accuracy of the TCABC algorithm is verified under different circumstances, and several recommendations are given for using the TCABC algorithm to detect structural damages under actual conditions. Finally, an experimental study is applied to examine the performance of TCABC algorithm for damage detection. The results show the following: (1) compared to traditional ABC algorithm, TCABC algorithm performs better; (2) fewer groups lead to faster convergence as demonstrated by both algorithms used in the same damage situation; (3) TCABC algorithm can infer the locations and extents of the damage when the groupings are inaccurate; (4) the accuracy of the field test data profoundly affects the precision of the damage detection results. In other words, stronger noises result in worse identification results; (5) whether or not the noises exist, the more data are measured, the more accurate the results can be achieved; (6) the TCABC algorithm can efficiently detect structural damage in the experimental study.