Deep learning has been widely adopted in the field of network traffic classification due to its unique advantages in handling encrypted network traffic. However, most existing deep learning models can only classify known encrypted traffic that has been sampled and labeled. In this paper, we propose CM-UTC, a cost-sensitive matrix-based method for classifying unknown encrypted traffic. CM-UTC explores the probability distribution of the DNN output layer to filter out the unknown classes and further designs a cost-sensitive matrix to address the class imbalance problem. Additionally, we propose the utilization of the Harris Hawk optimization algorithm to modify the model parameters and improve its performance. The experiments are validated on two different datasets, and the results demonstrate that CM-UTC not only outperforms existing methods in terms of overall performance but also exhibits superior capability in correctly identifying samples from the minority class.