The external appearance of an olive's skin is the most decisive factor in determining its quality as a fruit. This work tries to establish a hierarchical model based on the features extracted from images of olives reflecting their external defects. Seven commercial categories of olives, established by product experts, were used: undamaged olives, mussel-scale or 'serpeta', hail-damaged or 'granizo', mill or 'rehus', wrinkled olive or 'agostado', purple olive and undefined-damage or 'molestado'. The original images were processed using segmentation, colour parameters and morphological features of the defects and the whole fruits. The application of three consecutive discriminant analyses resulted in the correct classification of 97% and 75% of olives during calibration and validation, respectively. However the correct classification percentages vary greatly depending on the categories, ranging 80-100% during calibration and 38-100% during validation.