As one of the most common coupling elements in infrastructures, bolted joints play an important part in ensuring the integrity and safety of the whole system, whose failure may cause disastrous consequences. In recent years, precise detection and evaluation of bolt looseness have attracted numerous researchers' interest. However, the reliability of existing methods cannot be well guaranteed in long-term field detection, and real-time feedback is rather costly. This paper proposes a novel bolt looseness detection method based on audio recognition and deep learning. Firstly, a percussion experiment was designed to collect audio signals of bolts at different torque levels. Then, the time-domain bolt percussion signals were converted into Mel-frequency spectrograms, and the convolutional neural network (CNN) was adopted to mining deep information from the images for classification. To further verify the effect of different initial prestress levels on the vibration frequency of the bolted joint, a numerical study was conducted with the consideration of three different prestress levels. The results reveal that the proposed method has a high recognition accuracy in identifying bolt looseness conditions. Additionally, an iOS APP of acoustic vibration was established for real application. The prerecorded and untrained percussion audio was used to simulate the real-time bolt looseness detection, which shows its potential in real future applications.