Plant pests and diseases are a significant threat to almost all major types of plants and global food security. Traditional inspection across different plant fields is time-consuming and impractical for a wider plantation size, thus reducing crop production. Therefore, many smart agricultural practices are deployed to control plant diseases and pests. Most of these approaches, for example, use vision-based artificial intelligence (AI), machine learning (ML), or deep learning (DL) methods and models to provide disease detection solutions. However, existing open issues must be considered and addressed before AI methods can be used. In this study, we conduct a systematic literature review (SLR) and present a detailed survey of the studies employing data collection techniques and publicly available datasets. To begin the review, 1349 papers were chosen from five major academic databases, namely Springer, IEEE Xplore, Scopus, Google Scholar, and ACM library. After deploying a comprehensive screening process, the review considered 176 final studies based on the importance of the method. Several crops, including grapes, rice, apples, cucumbers, maize, tomatoes, wheat, and potatoes, have tested mainly on the hyperspectral imagery and vision-centered approaches. Support Vector Machines (SVMs) and Logistic regression (LR) classifiers demonstrated an increased accuracy in experiments compared to traditional classifiers. Besides the image taxonomy, disease localization is depicted in these approaches as a bottle neck to disease detection. Cognitive CNNs with attention mechanisms and transfer learning are showing an increasing trend. There is no standard model performance assessment though the majority use accuracy, recall, precision, F1 Score, and confusion matrix. The available 11 datasets are laboratory and in-field based, and 9 are publicly available. Some laboratory-based datasets are considerably small, making them impractical in experiments. Finally, there is a need to avail models with fewer parameters, implementable on small devices and large datasets accommodating several crops and diseases to have robust models.INDEX TERMS Convolutional neural networks, deep learning, image processing, machine learning, plant disease detections.