This paper investigates the application of eight color models for automatic palm oil Fresh Fruit Bunch (FFB) ripeness classification with multi-class Support Vector Machine (SVM). Ripeness classification is important during harvesting to ensure that they are harvested during the correct ripe stage for optimum oil production. Since color is a significant indicator for agriculturists to determine the ripeness of FFB, it is critical to determine the right color model. Eight color models have been investigated namely, HSV, I1I2I3, LAB, XYZ, YCbCr, YIQ, YUV and RGB. Color moments were extracted from each of these color models for the classification of four stages of FFB ripeness that are unripe, under-ripe, ripe and over-ripe. A database of five hundred images of palm oil FFB has been constructed and experiments showed that YCbCr and YUV outperform the other color models.