From ancient times olive tree cultivation has been one of the most crucial agricultural activities for Mediterranean countries. In recent years, the role of Artificial Intelligence in agriculture is increasing: its use ranges from monitoring of cultivated soil, to irrigation management, to yield prediction, to autonomous agricultural robots, to weed and pest classification and management, for example, by taking pictures using a standard smartphone or an unmanned aerial vehicle , and all this eases human work and makes it even more accessible.In this work, a method is proposed for olive disease classification, based on an adaptive ensemble of two EfficientNet-b0 models, that improves the state-of-the-art accuracy on a publicly available dataset by 1.6-2.6%. Both in terms of the number of parameters and the number of operations, our method reduces complexity roughly by 50% and 80%, respectively, that is a level not seen in at least a decade. Due to its efficiency, this method is also embeddable into a smartphone application for real-time processing.