Accurate and fast scale estimation of targets is a challenging research problem in visual object tracking. Most trackers employ an exhaustive scale search to estimate the target size. The exhaustive search strategy is computationally expensive and it struggles when encountered with large scale variations. In this paper, a scale adaptive method is proposed, which not only improves the tracking performance, but also greatly reduces the computational costs and improves the tracking speed. Based on the scale estimation method of SAMF, the original 7 fixed scale sizes were reduced to 3, and an adaptive scale size was added. Three fixed scales were used to determine the direction of scale change, and the APCE change rate of the current frame and the previous frame was used to control another adaptive scale size. Finally, the optimal scale estimation was determined. Additionally, we investigate the update strategy to further improve the tracking accuracy. Extensive experiments on OTB50, OTB100 and VOT-ST2019 datasets demonstrate that the proposed method can tackle challenging videos well compared with baseline tracker. On OTB, we obtain a gain of 7.0% in Distance Precision, and 18.8% in Centre Location Error on the selected 43 videos with scale variation attribute, and a mean gain of 6.2% in Precision and 4.6% in Success plots on OTB50, compared with the baseline tracker SAMF. Furthermore, the proposed approach improves the tracking speed by 34% in FPS compared with SAMF. INDEX TERMS Object tracking, correlation filter, scale estimation, model update.