Automated Traffic sign board classification system is one of the key technologies of Intelligent Transportation Systems (ITS). Traffic Surveillance System is being more and important with improving urban scale and increasing number of vehicles. This Paper presents an intelligent sign board classification method based on blob analysis in traffic surveillance. Processing is done by three main steps: moving object segmentation, blob analysis, and classifying. A Sign board is modelled as a rectangular patch and classified via blob analysis. By processing the blob of sign boards, the meaningful features are extracted. Tracking moving targets is achieved by comparing the extracted features with training data. After classifying the sign boards the system will intimate to user in the form of alarms, sound waves. The experimental results show that the proposed system can provide real-time and useful information for traffic surveillance.