Recently, Siamese-based trackers have become the primary research direction in single object tracking. However, a majority of these works aim at improving the backbone network of the tracker to boost its performance, thereby overlooking the significant impact that the template and the search region of the input to the tracker have on tracking accuracy. To address the aforementioned issues, we propose an Asymmetrical Transformer Tracker with Prior Templates (AtptTrack), which includes a tracking branch and a template update branch. The function of the tracking branch is to receive the image pairs and output the tracking results. In the template update branch, an updating strategy is employed to compute the cosine similarity between the input template and the tracking result. Based on this, four prior templates are generated, serving as essential supplementary features for the template. These prior templates are concatenated with the tracking results to get a hybrid template for subsequent tracking, enhancing the richness and accuracy of the template features. Furthermore, to further enrich the information content of the template and search region, we propose multi-scale embedding to process the input image pairs, which can enhance the completeness and continuity of the object features. Our tracker has been extensively tested on five benchmarks. The experiments demonstrate that our tracker achieves the state-of-the-art performance. Particularly on the OTB100 dataset, our tracker AtptTrack achieves an AUC score of 0.709, and it outperformed the second-place tracker in the deformation and occlusion challenges by 2.99% and 0.5%, respectively.