We study theoretically the groundstates of two-dimensional Bose-Hubbard models which are frustrated by gauge fields. Motivated by recent proposals for the implementation of optically induced gauge potentials, we focus on the situation in which the imposed gauge fields give rise to a pattern of staggered fluxes, of magnitude α and alternating in sign along one of the principal axes. For α = 1/2 this model is equivalent to the case of uniform flux per plaquette n φ = 1/2, which, in the hard-core limit, realizes the "fully frustrated" spin-1/2 XY model. We show that the mean-field groundstates of this frustrated Bose-Hubbard model typically break translational symmetry. We introduce a general numerical technique to detect broken symmetry condensates in exact diagonalization studies. Using this technique we show that, for all cases studied, the groundstate of the Bose-Hubbard model with staggered flux α is condensed, and we obtain quantitative determinations of the condensate fraction. We discuss the experimental consequences of our results. In particular, we explain the meaning of gauge-invariance in ultracold atom systems subject to optically induced gauge potentials, and show how the ability to imprint phase patterns prior to expansion can allow very useful additional information to be extracted from expansion images.