Ripe fruit offer readily available nutrients for many animals, including fruit fly larvae (Diptera: Tephritidae) and their associated rot-inducing bacteria. Yet, during most of their ontogeny, fruit remain chemically defended and effectively suppress herbivores and pathogens by high levels of secondary metabolites. Olive flies (Bactrocera oleae) are uniquely able to develop in unripe olives. Unlike other frugivorous tephritids, the larvae maintain bacteria confined within their midgut caeca. We examined the interaction between larvae, their associated bacteria, and fruit chemical defence, hypothesizing that bacterial contribution to larval development is contingent on the phenology of fruit defensive chemistry. We demonstrate that larvae require their natural complement of bacteria (Candidatus Erwinia dacicola: Enterobacteriaceae) in order to develop in unripe olives. Conversely, when feeding on ripe fruit, larval development proceeds independently of these bacteria. Our experiments suggest that bacteria counteract the inhibitory effect of oleuropein—the principal phenolic glycoside in unripe olives. In light of these results, we suggest that the unique symbiosis in olive flies, compared with other frugivorous tephritids, is understood by considering the relationship between the fly, bacteria and fruit chemistry. When applied in an evolutionary context, this approach may also point out the forces which shaped symbioses across the Tephritidae.