Multi-object tracking is an important branch of computer vision, which is mostly used for behavior recognition and event analysis. At present, most of the research focuses on the accuracy of tracking. However, the real-time performance is also urgently desired but there is lack of research. As a result, Deepsort, proposed 5 years ago, is still the most widely used tracker in real applications. In this paper, a Bilateral Association Tracking (BAT) framework is proposed. It uses tracklet as the basic node instead of discrete detection for tracking. Meanwhile, a Parzen density based Hierarchical Agglomerative Clustering (P-HAC) algorithm is introduced to describe the density distribution of targets and generate tracklets with high confidence. In addition, Dual Appearance Features (DAF) is proposed which considers both spatial and temporal features of tracklets and promotes the accuracy of tracklet association. Experiments are conducted on popular benchmarks such as MOT2017, Visdrone and KITTI. BAT outperforms Deepsort on both association accuracy and trajectory integrity without obvious efficiency decline. Compared with other state-of-the-art trackers, BAT shows significant advantage on computational cost while performing competitive tracking accuracy as well. It is hoped that the research can promote the applications on real-time tracking in the near future.