With the rapid increase in the world’s population, there is an ever-growing need for a sustainable food supply. Agriculture is one of the pillars for worldwide food provisioning, with fruits and vegetables being essential for a healthy diet. However, in the last few years the worldwide dispersion of virulent plant pests and diseases has caused significant decreases in the yield and quality of crops, in particular fruit, cereal and vegetables. Climate change and the intensification of global trade flows further accentuate the issue. Integrated Pest Management (IPM) is an approach to pest control that aims at maintaining pest insects at tolerable levels, keeping pest populations below an economic injury level. Under these circumstances, the early identification of pests and diseases becomes crucial. In this work, we present the first step towards a fully fledged, semantically enhanced decision support system for IPM. The ultimate goal is to build a complete agricultural knowledge base by gathering data from multiple, heterogeneous sources and to develop a system to assist farmers in decision making concerning the control of pests and diseases. The pest classifier framework has been evaluated in a simulated environment, obtaining an aggregated accuracy of 98.8%.