Olive (Olea europaea L.) is an important Mediterranean tree species with a longstanding history of cultivation, boasting a diverse array of local cultivars. While traditional olive orchards are valued for their cultural and aesthetic significance, they often face economic sustainability challenges in the modern context. The success of both traditional and newly introduced cultivars (e.g. those obtained by crossbreeding) is hindered by self-incompatibility, a prevalent issue for this species that results in low fruit set when limited genetic diversity is present. Further, biological, environmental, and agronomic factors have been shown to interlink in shaping fertilization patterns, hence impacting on the final yield. Climatic conditions during pollination, such as excessive rainfall or high temperatures, can further exacerbate the problem. In this work, we provide an overview of the various complex and multifaceted factors that trigger the phenomenon of sub-optimal fruit set in olive trees. Through this work, we aim to provide a comprehensive understanding of the interplay among these factors, shedding light on potential mechanisms and pathways that contribute to the observed outcomes in the context of self-incompatibility of olive. This review aims to contribute to the development of sustainable olive production systems and the preservation of this vital component of Mediterranean culture.