Economic, social, and climatic conditions affect agricultural production. Those changes are relevant to the rainfed agricultural areas of the Mediterranean Belt, including Spain—the largest producer of olive oil in the world. However, little is known about the effect of the climate on olive production and farmer income. In this study, the correlation between changes in rainfall and total olive production was examined using a long-term dataset (28 years) on conventional rainfed production and tillage soil management. The dataset focused on different olive groves in the municipality of Bailén (Jaén, Andalusia) that have been owned by the same farmer since 1966. The province of Jaén is the region of Spain with the highest production of olive oil and the largest area of olive groves. The data included annual rainfall, production per plot and the price of olives. After calculating missing data to complete the rainfall series, pairwise correlation analysis with nonparametric Spearman's rank coefficients and principal component analysis were used to process the data. The results showed that higher production coincided with increased rainfall during August and December. Therefore, we concluded that the impact of rainfall on olive production is variable and depends on drought intensity and the monthly rainfall distribution. An economic study showed that farmer income was highly dependent on the seasonal distribution of the rainfall among other factors such as the price of olives. Farmer income was low during drought periods, indicating that rainfed agriculture is perceived by farmers as unsustainable due to the resulting highly variable income. This study could help to prevent risks to food security in the future. We recognise that other key factors have also been important influences on the fluctuations in olive production over the years, such as soil properties and plant status. However, cultivating olives without irrigation—depending only on the total rainfall amount and rainfall intensity to supply all of the water consumed by the plants—is very risky too. This research demonstrates that the subsistence of Mediterranean rainfed olive farmers can be highly dependent on the rainfall conditions.