Methods and techniques for Intelligent Transport Systems (ITS) are actively researched recently. Road sign detection and recognition, in particular, have attracted an increasing interest over the last decade. An apparent shape of a road sign is deformed depending on the relative orientation and distance between a driver and the road sign, and the sign itself might be occluded by other obstacles. In this paper, we propose a novel method for classifying polygonal road signs invariant to these environments. In the detection stage the proposed system first specifies the image region of a road sign by using both color information and its marginal distribution. Determining successively the outmost edges and vertexes of the road signs detected provides the parameters of geometric deformation with respect to the apparent shape of the signs. The template patterns to be matched are then deformed in the recognition stage on the basis of the estimated geometric parameters. The detected signs are finally classified by using Phase-Only Correlation(POC) function around previously specified feature points. According to experimental results, the detection rate is 96.1% and recognition rate is 90.1% for 197 images taken under various illumination conditions.