Plant leaves and crops play a crucial role as a primary food source globally, making significant contributions to dietary iron intake (9%) and energy consumption (23%) per capita in the Asian region. Bacterial, yeast, and other microbial diseases pose significant challenges to farmers as they detrimentally impact plant health and reduce crop productivity. The manual diagnosis of these diseases poses a considerable challenge, particularly in regions with a scarcity of professionals specializing in leaves and crop protection. Automating leaf disease detection and providing easily accessible decision-support resources are crucial for facilitating efficient leaf protection strategies and mitigating crop damage. Despite multiple classification methods for diagnosing leaf diseases, a secure and accurate approach that fulfills these requirements has not yet been identified. This paper presents an architectural framework called Lightweight Federated Transfer Learning (LFTL) that addresses the challenge of Leaf Disease Detection and Classification (LDDC) while ensuring data privacy limitations are upheld. A dataset consisting of leaf disease images has been compiled, characterized by an imbalance in the distribution of the diseases. The collection includes four conditions: bacterial decay, brown spot, blast, and tungro, corresponding image counts of 1695, 1551, 1711, and 1419, respectively. Following the preprocessing stage, the LFTL framework was tested using both Independent and Identically Distributed (IID) and non-IID datasets. The study commenced with an efficacy evaluation of the Convolutional Neural Network (CNN) and eight TL models in the LDDC. The framework’s performance was evaluated across different circumstances and compared to conventional and federated learning models. The study’s findings revealed that the LFTL framework outperformed traditional distributed deep-learning classifiers, thus demonstrating its efficacy in individual and multiple client scenarios.