Convolutional Neural Networks (CNNs) have been established as a powerful class of models for image recognition problems. Despite their success in other areas, CNNs have been applied only for very limited agricultural applications due to the need for large datasets. The aim of this research is to design a robust CNN model that classifies raw coffee beans into their 12 quality grades using small datasets which have high data variability. The dataset contains images of raw coffee beans acquired in two sets using different acquisition technique under varying illuminations which poses a complex challenge to designing a robust model. To design the model, preprocessing techniques were applied to the input in order to reduce task irrelevant features. But adding the preprocessing techniques did not improve the performance of the CNN model for our dataset. We have also used ensemble methods to solve the high variance that exists in networks when working with small datasets. Finally, we were able to design a model that classifies the beans into their quality grades with an accuracy of 89.01% on the test dataset.