The cultivated olive (Olea europaea L. subsp. europaea var. europaea) is one of the most valuable fruit trees worldwide. However, the hormonal mechanisms underlying the fruit growth and ripening in olives remain largely uncharacterized. In this study, we investigated the physiological and hormonal changes, by ultra-high performance liquid chromatography-mass spectrometry (UHPLC-MS), as well as the expression patterns of hormone-related genes, using quantitative real-time PCR (qRT-PCR) analysis, during fruit growth and ripening in two olive cultivars, ‘Arbequina’ and ‘Picual’, with contrasting fruit size and shape as well as fruit ripening duration. Hormonal profiling revealed that olive fruit growth involves a lowering of auxin (IAA), cytokinin (CKs), and jasmonic acid (JA) levels as well as a rise in salicylic acid (SA) levels from the endocarp lignification to the onset of fruit ripening in both cultivars. During olive fruit ripening, both abscisic acid (ABA) and anthocyanin levels rose, while JA levels fell, and SA levels showed no significant changes in either cultivar. By contrast, differential accumulation patterns of gibberellins (GAs) were found between the two cultivars during olive fruit growth and ripening. GA1 was not detected at either stage of fruit development in ‘Arbequina’, revealing a specific association between the GA1 and ‘Picual’, the cultivar with large sized, elongated, and fast-ripening fruit. Moreover, ABA may play a central role in regulating olive fruit ripening through transcriptional regulation of key ABA metabolism genes, whereas the IAA, CK, and GA levels and/or responsiveness differ between olive cultivars during olive fruit ripening. Taken together, the results indicate that the relative absence or presence of endogenous GA1 is associated with differences in fruit morphology and size as well as in the ripening duration in olives. Such detailed knowledge may be of help to design new strategies for effective manipulation of olive fruit size as well as ripening duration.