In the olive (Olea europaea L.), an economically leading oil crop worldwide, fruit size and yield are determined by the early stages of fruit development. However, few detailed analyses of this stage of fruit development are available. This study offers an extensive characterization of the various processes involved in early olive fruit growth (cell division, cell cycle regulation, and cell expansion). For this, cytological, hormonal, and transcriptional changes characterizing the phases of early fruit development were analyzed in olive fruit of the cv. ‘Picual’. First, the surface area and mitotic activity (by flow cytometry) of fruit cells were investigated during early olive fruit development, from 0 to 42 days post-anthesis (DPA). The results demonstrate that the cell division phase extends up to 21 DPA, during which the maximal proportion of 4C cells in olive fruits was reached at 14 DPA, indicating that intensive cell division was activated in olive fruits at that time. Subsequently, fruit cell expansion lasted as long as 3 weeks more before endocarp lignification. Finally, the molecular mechanisms controlling the early fruit development were investigated by analyzing the transcriptome of olive flowers at anthesis (fruit set) as well as olive fruits at 14 DPA (cell division phase) and at 28 DPA (cell expansion phase). Sequential induction of the cell cycle regulating genes is associated with the upregulation of genes involved in cell wall remodeling and ion fluxes, and with a shift in plant hormone metabolism and signaling genes during early olive fruit development. This occurs together with transcriptional activity of subtilisin-like protease proteins together with transcription factors potentially involved in early fruit growth signaling. This gene expression profile, together with hormonal regulators, offers new insights for understanding the processes that regulate cell division and expansion, and ultimately fruit yield and olive size.