Cacao pod's ideal harvesting time is when it is about to be ripe. Immature harvest would result in hard cacao beans not suitable for fermentation, while overripe cacao pods lead to fungal-infected, defective, and poor-quality yields. The demand for high-quality cacao products is expected to rise due to advancing technology in the present. Pre-harvesting needs to provide optimal identification of which amongst the pods are ripened enough and ready for the next stage of the cacao process. This paper recommends a technique to determine the ripeness of cacao. Nine hundred thirty-three cacao samples were used to collect thumping audio data at five different pod's exocarp locations. Each sound file is 1 second long, creating 4665 cacao sound file datasets at 16kHz sample rate and 16-bit audio bit depth. The process of the Mel-Frequency Cepstral Coefficient Spectogram was then applied to extract recognizable features for the training process. The deep learning method integrated was a convolutional neural network (CNN) to classify the cacao sound successfully. The experimental design model's output exhibits an accuracy of 97.50 % for the training data and 97.13 % for the validation data. While the overall accuracy mean of the classification system is 97.46 %, whether the cacao is unripe or ripe.