Traditional data-driven intelligent fault diagnosis methods have been successfully developed under the closed set assumption (CSA). CSA-based fault diagnosis assumes that the fault types in the test set are consistent with that in the training set, which can achieve high accuracy, but this is generally not valid in real-world industrial applications where the collection of data in industrial applications is often limited. As it is unrealistic to assume that the training set will cover all fault types, the application of the fault classifier may fail when the test set contains unknown fault types because the probability of input samples belonging to unknown types cannot be obtained. To solve the problem of how unknown fault types may be accurately identified, this paper further studies the open set assumption (OSA) fault diagnosis. We propose an open set convolutional neural network (OS-CNN) method and apply our OS-CNN model to an improved OpenMax method as a deep network to accurately detect unknown fault types. The overall performance was significantly improved as our OS-CNN model was able to effectively tighten the boundary of known classes and limit the open-space risk for the OpenMax method based on distance modeling. The overall effectiveness of the proposed method was verified by experimental studies based on four different bearing datasets. Compared with state-of-the-art OSA fault diagnosis method, our method cannot only realize the correct classification of the known fault classes, but it can also accurately detect the unknown fault classes.