Linear accelerators capable of delivering high brightness electron beams are essential components of a number of research tools, such as free electron lasers (FELs) and elementary particle colliders. In these facilities the charge density is high enough to drive undesirable collective effects (wakefields) that may increase the beam emittance relative to the injection level, eventually degrading the nominal brightness. We formulate a limit on the final electron beam brightness, imposed by the interplay of geometric transverse wakefield in accelerating structures and coherent synchrotron radiation in energy dispersive regions. Numerous experimental data of vacuum ultraviolet and x-ray FEL drivers validate our model. This is then used to show that a normalized brightness of $10 16 A=m 2 , promised so far by ultralow charge beams ($ 1-10 pC), can in fact be reached with a 100 pC charge beam in the 1.2 GeV FERMI@Elettra accelerator, with the existing machine configuration.