As a composite material, the stability of rock mass is usually controlled by a joint. During the process of excavation, the normal stress of the joint decreases continuously, and then the shear strength of the joint decreases, which may eventually lead to the instability and failure of rock mass. Previous studies have mainly focused on the shear behavior of joints under constant normal stress, but have rarely considered the unloading of normal stress. In this paper, a direct shear test of joints with different roughness was carried out, in which the shear stress remained unchanged while the normal stress decreased. The strength characteristics of joints were explored, and the deformation and acoustic emission-counting characteristics of joints were analyzed by digital image correlation (DIC) techniques and acoustic emission (AE). A new method for predicting the instability of joints under normal unloading was proposed based on the evolution law of normal deformation energy (Un), tangential deformation energy (Us) and total deformation energy (U0). The results show the following: (1) The unloading amount of normal stress was enlarged for greater initial normal stress and roughness, while it decreased with an increase in initial shear stress. (2) AE events reached their maximum when the normal stress was equal to the failure normal stress, and the b-value fluctuated more frequently in stable development periods under normal unloading conditions. (3) U0 would change with the loading and unloading of stress, and this may be used to predict the unloading instability of rock mass using the abrupt change of U0.