Scene image classification systems firstly need to locate the objects, and then classify the whole image. The color feature is importance to describe the properties of an image surface. The paper presents a framework for scene images to label local regions using color features. The paper uses maker-controlled watershed algorithm to segment the input image into regions. This paper uses the segmented regions as a basic input unit, and then extract Color Histogram (CH) and Color Moment (CM) features in HSV space. This system performs labeling using 3-layer Feed Forward Neural Network (FFNN) classifier. The system tests accuracy on public Microsoft Research Cambridge (MSRC) 9-class dataset.