Identifying a device and detecting a change in its position is critical for secure devices management in the Internet of Things (IoT). In this paper, a device management system is proposed to track the devices by using audio-based location distinction techniques. In the proposed scheme, traditional cryptographic techniques, such as symmetric encryption algorithm, RSA-based signcryption scheme, and audio-based secure transmission, are utilized to provide authentication, non-repudiation, and confidentiality in the information interaction of the management system. Moreover, an audio-based location distinction method is designed to detect the position change of the devices. Specifically, the audio frequency response (AFR) of several frequency points is utilized as a device signature. The device signature has the features as follows. (1) Hardware Signature: different pairs of speaker and microphone have different signatures; (2) Distance Signature: in the same direction, the signatures are different at different distances; and (3) Direction Signature: at the same distance, the signatures are different in different directions. Based on the features above, a movement detection algorithm for device identification and location distinction is designed. Moreover, a secure communication protocol is also proposed by using traditional cryptographic techniques to provide integrity, authentication, and non-repudiation in the process of information interaction between devices, Access Points (APs), and Severs. Extensive experiments are conducted to evaluate the performance of the proposed method. The experimental results show that the proposed method has a good performance in accuracy and energy consumption.