The agriculture sector in Jordan depends very much on planting the olive trees. More than ten million of olive trees are planted in the Jordanian soil. Olive fruit are harvested for two purposes; either to produce oil or to produce olive table (pickled olive). Olive fruit harvesting time for extracting the oil from the olive fruit is crucial. Hence, harvesting the olive fruit on ripping time gives the best amount and quality of oil. It also, could lose 15% to 20% of multiple values because of harvesting time. Olive fruit ripping time is varied since it depends on the rainfall, temperature and cultivation. A system to predict the optimal time for harvesting olive fruit for producing oil only is introduced. It based one Digital Image Processing (DIP) and artificial intelligent neural network. Moreover, four features were extracted from the olive fruit image based on the red, green and blue colors. The proposed system tested olive fruits in three stages of ripping time; under ripping, on ripping and over ripping. The classification accuracy achieved in the three stages was 97.51% in under ripping stage 95.10% in ripping stage, and 96.12% in over ripping stage. The proposed system performance was 96.14%.