The olive fruit fly can damage up to 100% of the harvested fruit and can cause up to 80% reduction of the value of the resulting olive oil. Therefore, it is important to early detect its presence in the olive orchard to take the appropriate chemical or biological countermeasures as early as possible. Traps filled with attractant pheromones are typically deployed across the orchard to attract and capture the flies. Traditionally, the captured flies were manually counted which is error prone. Recently, the traps are employed with cameras and communication devices to send pictures of the captured flies to experts for analysis which is also error prone and inefficient. Consequently, machine and deep learning have been exploited to develop fully automated and accurate detection that does not include human in the loop. Such a learning problem is challenging due to the small size of the detected object, the differences in the light conditions at which pictures were taken, and the lack of enough data to train the learning model. In this paper, we present a deep learning framework for detecting and counting the number of olive fruit flies that exploits data augmentation to increase the dataset size, includes negative samples in the training to improve the detection accuracy, and normalizes the images to the color of the trap background, i.e., yellow, to unify the illumination conditions. The results of the proposed framework show a precision of 0.84, a recall of 0.97, an F1-score of 0.9 and mean Average Precision (mAP) of 96.68% which significantly outperforms existing pest detection systems.