The uncertain dynamics and environmental impacts in collision avoidance planning are always the key issues of autonomous navigation for unmanned surface vessel, due to the strong interactive ability and learning ability of deep learning, the combination of CNN and LSTM provide relevance and memory characteristics, which can extract the potential influence features of collision avoidance and correlation of historical strategies, to gain the more general trend of collision avoidance. Meanwhile, to make sure the real-time problem, the heuristic planning based on improved Electromagnetism-like mechanism algorithm is utilized to seek the optimal strategy with the rolling iteration of CNN–LSTM structure, for obtaining a simple and effective on-line scheme of collision avoidance. The complete algorithm has been tested and validated under the different situation comparing with the deterministic algorithm, it is worth mentioning that the algorithm can potentially be developed to advance collision-free planning especially in multiple vessels situation.