Domain adaptation techniques have effectively tackled fault diagnosis under varying operational conditions. Many existing studies presume that machine health states remain consistent between training and testing data. However, in real-world scenarios, fault modes during testing are often unpredictable, introducing unknown faults that challenge the effectiveness of domain adaptation-based fault diagnosis methods. To address these challenges, this paper proposes a Deep Open Set Domain Adaptation Network (DODAN). Firstly, a feature extraction module based on multi-scale depthwise separable convolutions is constructed for discriminative feature extraction. To improve the model’s adaptability, an adversarial training strategy is implemented to learn generalized features that are resilient to unknown domain shifts. Additionally, an outlier detection module is employed to determine the optimal decision boundaries for each class representation space, enabling the classification of known fault modes and the identification of unknown ones. Extensive diagnostic experiments on two marine machinery datasets validate the effectiveness of the proposed method. Furthermore, ablation studies verify the efficacy of the proposed modules and strategies, highlighting significant potential for practical applications.