We address the problem of automatically analyzing hockey scenes by estimating the panning, tilting and zooming parameters of the broadcasting cameras, tracking hockey players in these scenes, and constructing a visual description of the scenes as trajectories of those players. Given quite fast and non-smooth camera motions to capture highly complex and dynamic scenes of hockey, tracking hockey players that are small blob-like, non-rigid and amorphous becomes an extremely difficult task. We suggest a new method of automatically computing the mappings to represent the globally consistent map of the hockey scenes by removing camera motions, and implement a colorbased sequential Monte Carlo tracker to track hockey players to estimate their real world position on the rink. The result demonstrates a quite successful performance on both objectives. We make two new contributions in this research. First, we introduce a new model fitting algorithm to reduce projection errors. Second, we use an adaptive model to improve the current state-of-art color-based probablistic tracker. Our approach is also applicable for video annotation in other sports, surveillance, or many other situations that require obect tracking on a planar surface. Since there have not been any hockey annotation systems developed in the past, we hope that our system would become a stepping stone for automatic video annotation in hockey.