This paper presents a novel color quantization algorithm for compressed video data. The proposed algorithm extracts discrete cosine (DC) transform coefficients and motion vectors of blocks in a shot to estimate a cumulative color histogram of the shot and, based on the estimated histogram, design a color palette for displaying the video sequence in the shot. It significantly reduces the complexity of the generation of a palette by effectively reducing the number of training vectors used in training a palette without sacrificing the quality. The palette obtained can provide a good display quality even if zooming and panning exist in a shot. The experimental results show that the proposed method can achieve a significant signal-to-noise ratio improvement as compared with conventional video color-quantization schemes when zooming and panning are encountered in a shot.