Ripe oil palm fresh fruit bunch allows extraction of high-quality crude palm oil and kernel palm oil. As the fruit ripens, its surface color changes from black (unripe) or dark purple (unripe) to dark red (ripe). Thus, the surface color of the oil palm fresh fruit bunches may generally be used to indicate the maturity stage. Harvesting is commonly done by relying on human graders to harvest the bunches according to color and number of loose fruits on the ground. Non-destructive methods such as image processing and computer vision, including object detection algorithms have been proposed for the ripeness classification process. In this paper, several object detection algorithms were investigated to classify the ripeness of oil palm fresh fruit bunch. MobileNetV2 SSD, EfficientDet (Lite0, Lite1 and Lite2) and YOLOv5 (YOLOv5n, YOLOv5s and YOLOv5m) were simulated and compared in terms of their mean average precision, recall, precision and training time. The models were trained on a dataset with four main ripeness classes: ripe, unripe, half-ripe, and over-ripe. In conclusion, object detection algorithms can be used to classify different ripeness levels of oil palm fresh fruit bunch, and among the different models, YOLOv5m showed promising results with a mean average precision of 0.842 (0.5:0.95).