For a typical surface automated visual inspection (AVI) instrument of planar materials, defect classification is an indispensable part after defect detection, which acts as a crucial precondition for achieving the on-line quality inspection of end products. In the industrial environment of manufacturing flat steels, this task is awfully difficult due to diverse defect appearances, ambiguous intraclass and interclass distances. This paper attempts to present a focused but systematic review of the traditional and emerging automated computer-vision-based defect classification methods by investigating approximately 140 studies on three specific flat steel products of con-casting slabs, hot-rolled steel strips and cold-rolled steel strips. According to the natural image processing procedure of defect recognition, the diverse approaches are grouped into five successive parts: image acquisition, image preprocessing, feature extraction, feature selection and defect classifier. Recent literature has been reviewed from an industrial goal-oriented perspective to provide some guidelines for future studies, as well as to recommend suitable methods for boosting the surface quality inspection level of AVI instruments.