With a wide use of AIS data in maritime transportation, there is an increasing demand to develop algorithms to efficiently classify a ship's AIS data into different movements (static, normal navigation and manoeuvring). To achieve this, several studies have been proposed to use labeled features but with the drawback of not being able to effectively extract the details of ship movement information. In addition, a ship movement is in a free space, which is different to a road vehicle's movement in road grids, making it inconvenient to directly migrate the methods for GPS data classification into AIS data. To deal with these problems, a Convolutional Neural Network-Ship Movement Modes Classification (CNN-SMMC) algorithm is proposed in this paper. The underlying concept of this method is to train a neural network to learn from the labeled AIS data, and the unlabeled AIS data can be effectively classified by using this trained network. More specifically, a Ship Movement Image Generation and Labelling (SMIGL) algorithm is first designed to convert a ship's AIS trajectories into different movement images to make a full use of the CNN's classification ability. Then, a CNN-SMMC architecture is built with a series of functional layers (convolutional layer, max-pooling layer, dense layer etc.) for ship movement classification with seven experiments been designed to find the optimal parameters for the CNN-SMMC. Considering the imbalanced features of AIS data, three metrics (average accuracy, 1 score and Area Under Curve (AUC)) are selected to evaluate the performance of the CNN-SMMC. Finally, several benchmark classification algorithms (K-Nearest Neighbors (KNN), Support Vector Machine (SVM) and Decision Tree (DT)) are selected to compare with CNN-SMMC. The results demonstrate that the proposed CNN-SMMC has a better performance in the classification of AIS data.