Several research works on disease detection in coffee plants have been presented in recent years. Leaf miner and rust are the most prevalent diseases in Arabica coffee plants. Early detection of such diseases allows farmer to take diagnostic actions before the infection spreads to neighboring plants. With advancements in drones and artificial intelligence (AI), the automatic detection of leaf diseases is gaining prominence in the field of smart agriculture. Furthermore, it is critical to develop an accurate method for infestation detection with minimal computational complexity. Existing works for plant disease detection utilize pre-trained deep learning models with millions of parameters.A feasible trade-off has to be attained between accuracy and computational complexity for the deployment of such deep networks. This research proposes an effective method for disease detection in Arabica coffee plants using EfficientNetB0 architecture. The architecture of the EfficientNetB0 network was improvised by including a ghost module at its end. This integration allows the network to learn effectively with minimal parameters without compensating for the end accuracy. The proposed model has a total of 4,874,531 parameters which is significantly lesser than most of the state-of-the-art deep learning architectures and achieved an accuracy of 84%.