Image segmentation decomposes a given image into segments, i.e. regions containing "similar" pixels, that aids computer vision applications such as face, medical, and fingerprint recognition as well as scene characterization. Effective segmentation requires domain knowledge or strategies for object designation as no universal segmentation algorithm exists. In this paper, we propose a holistic framework to perform image segmentation in color space. Our approach unifies the linear smoothing filter, a similarity calculation in selected color space, and a clustering game model with various evolution dynamics. In our framework, the problem of image segmentation can be considered as a "clustering game". Within this context, the notion of a cluster turns out to be equivalent to a classical equilibrium concept from game theory, as the game equilibrium reflects both the internal and external cluster conditions. Experiments on image segmentation problems show the superiority of the proposed clustering game based image segmentation framework (CGBISF) using both the Berkeley segmentation dataset and infrared images (for which, we need to perform color fusion first) in autonomy, speed, and efficiency.