As the noise reduction performance of submarines continues to improve, it is difficult to detect and track submarines through acoustic detection techniques. Therefore, nonacoustic submarine detection techniques are becoming more and more important. The submarine movement will leave a wake vortex, and the information of the wake vortex can be used to invert the maneuvering state of the submarine. However, the wake vortex is constantly dissipated in the evolution process, and the strength of the wake vortex is constantly reduced, resulting in the gradual weakening of the characteristics of the wake vortex, which makes the inversion of submarine operating state difficult and less accurate. In order to solve the above problems, this paper proposes an improved wake vortex-based inversion method for submarine maneuvering state. Firstly, a random finite set of submarine wake vortex observation features is established to obtain the feature with the highest correlation degree with submarine maneuvering state in the random finite set. Secondly, the multiscale fusion module and attention mechanism are used to re-encode the weak features of the wake vortex image, and the salient features of the wake vortex image are extracted. Finally, the manipulation state of the wake vortex image is retrieved by the extracted salient features. The experimental results show that the average inversion accuracy of the proposed algorithm is improved by 1.27% in terms of manipulating state inversion of weak feature wake vortex images. The algorithm in this paper can realize the inversion of submarine maneuvering state in the case of weak submarine wake vortex image features and incomplete feature information. It provides the basis for the detection technology based on the submarine wake characteristics.