The quality of palm oil is strongly influenced by the maturity level of the fruit to be processed into palm oil. Many studies have been carried out for detecting and classifying the maturity level of oil palm fruit to improve the quality with the use of computer vision. However, most of these studies use datasets in the form of images of oil palm fresh fruit bunches (FFB) with incomplete categorization according to real conditions in palm oil mills. Therefore, this study introduces a new complete dataset obtained directly from palm oil mills in the form of videos and images with different categories in accordance with the real conditions faced by the grading section of the palm oil mill. The video dataset consists of 45 videos with a single category of FFB videos and 56 videos with a collection of FFB with multiple categories for each video. Videos are collected using a smart phone with a size of 1280 × 720 pixels with .mp4 format. In addition, this dataset has also been annotated and labelled based on the maturity level of oil palm fruit with 6 categories, which are unripe, under-ripe, ripe, overripe, empty bunches and abnormal fruit.