Computer vision technology can automatically detect and recognize objects on the ground or on a court, such as balls, players, and lines, using camera sensors. These are non-contact sensors, which do not interfere with an athlete’s movement. The game elements detected by such measuring equipment can be used for game analysis, judgment, context recognition, and visualization. This paper proposes a method to automatically track the position of stones in curling sport images using computer vision technology. The authors extract the optimal feature vector of the mean-shift tracking algorithm by obtaining the optimal histogram from the color and edge information of the curling stone, thereby adaptively controlling the number of bins in the histogram. After evaluating the performance of the curling stone tracking method among 1424 image frames from curling sport videos, the authors found that the proposed method improved detection rate (overlap threshold = 0.9) by 14.85% compared to the general mean-shift method.