The implementation of intelligent technology in agriculture is seriously investigated as a way to increase agriculture production while reducing the amount of human labor. In agriculture, recent technology has seen image annotation utilizing deep learning techniques. Due to the rapid development of image data, image annotation has gained a lot of attention. The use of deep learning in image annotation can extract features from images and has been shown to analyze enormous amounts of data successfully. Deep learning is a type of machine learning method inspired by the structure of the human brain and based on artificial neural network concepts. Through training phases that can label a massive amount of data and connect them up with their corresponding characteristics, deep learning can conclude unlabeled data in image processing. For complicated and ambiguous situations, deep learning technology provides accurate predictions. This technology strives to improve productivity, quality and economy and minimize deficiency rates in the agriculture industry. As a result, this article discusses the application of image annotation in the agriculture industry utilizing several deep learning approaches. Various types of annotations that were used to train the images are presented. Recent publications have been reviewed on the basis of their application of deep learning with current advancement technology. Plant recognition, disease detection, counting, classification and yield estimation are among the many advancements of deep learning architecture employed in many applications in agriculture that are thoroughly investigated. Furthermore, this review helps to assist researchers to gain a deeper understanding and future application of deep learning in agriculture. According to all of the articles, the deep learning technique has successfully created significant accuracy and prediction in the model utilized. Finally, the existing challenges and future promises of deep learning in agriculture are discussed.