Player detection in sports video is a challenging task: In contrast to typical surveillance applications, a pan-tilt-zoom camera model is used. Therefore, simple background learning approaches cannot be used. Furthermore, camera motion causes severe motion blur, making gradient based approaches less robust than in settings where the camera is static. The contribution of this paper is a sequence adaptive approach that utilizes color information in an unsupervised manner to improve detection accuracy. Therefore, different color features, namely color histograms, color spatiograms and a color and edge directivity descriptor are evaluated. It is shown that the proposed color adaptive approach improves detection accuracy. In terms of maximum F1 score, an improvement from 0.79 to 0.81 is reached using block-wise HSV histograms. The average number of false positives per image (FPPI) at two fixed recall levels decreased by approximately 23%