A compact high-frequency surface wave radar, used for target detection, suffers from a low signal-to-noise ratio, low detection probability, a high false alarm rate, and low positioning accuracy; this is due to its low transmit power and the reduced aperture size of the receiving antenna array. When target tracking algorithms are applied to compact high-frequency surface wave radar data, track fragmentation often occurs and a long track may be broken into several track segments (a.k.a. tracklets), which degrade the tracking continuity for a maritime surveillance system. We present a multi-stage vessel tracklet association method, based on bidirectional prediction and optimal assignment, to associate the broken tracklets belonging to the same target, and connect them to form one continuous track in a multi-target tracking scenario. Firstly, two global motion parameters, i.e., the average heading and average speed, were, respectively, extracted from the newly initiated and terminated tracklets as features for a rough tracklet association, then k-means clustering was used to produce the preliminary tracklet pairs. Subsequently, the temporal and spatial constraints on the initiated and terminated tracklets were considered to refine the preliminary tracklet pairs, to obtain the candidate tracklet pairs. Finally, the tracklet association costs were calculated using Doppler velocity, range, and azimuth to determine the similarity between tracklets in the candidate tracklet pairs, and an association cost matrix was obtained. Then an optimal assignment method based on Jonker–Volgenant–Castanon algorithm was applied to the association matrix to achieve optimal tracklet matching by minimizing the total association costs. Tracklet association experiments with both simulated and field data were conducted; experimental results show that, compared with existing track segment association methods, the association accuracy of the proposed method is significantly improved with better tracking continuity.