Deep learning-based plant disease detection has gained significant attention from the scientific community. However, various aspects of real horticultural conditions have not yet been explored. For example, the disease should be considered not only on leaves, but also on other parts of plants, including stems, canes, and fruits. Furthermore, the detection of multiple diseases in a single plant organ at a time has not been performed. Similarly, plant disease has not been identified in various crops in the complex horticultural environment with the same optimized/modified model. To address these research gaps, this research presents a dataset named NZDLPlantDisease-v1, consisting of diseases in five of the most important horticultural crops in New Zealand: kiwifruit, apple, pear, avocado, and grapevine. An optimized version of the best obtained deep learning (DL) model named region-based fully convolutional network (RFCN) has been proposed to detect plant disease using the newly generated dataset. After final the most suitable DL model, data augmentation techniques were successively evaluated. Subsequently, the effects of image resizers with interpolators, weight initializers, batch normalization, and DL optimizers were studied. Finally, performance was enhanced by empirical observation of position-sensitive score maps and anchor box specifications. Furthermore, the robustness/practicality of the proposed approach was demonstrated using a stratified k-fold cross-validation technique and testing on an external dataset. The final mean average precision of the RFCN model was found to be 93.80%, which was 19.33% better than the default settings. Therefore, this research could be a benchmark step for any follow-up research on automatic control of disease in several plant species.