The formation of cold gas giants similar to Jupiter and Saturn in orbit and mass is a great challenge for planetesimal-driven core accretion models because the core growth rates far from the star are low. Here we model the growth and migration of single protoplanets that accrete planetesimals and gas. We integrated the core growth rate using fits in the literature to N-body simulations, which provide the efficiency of accreting the planetesimals that a protoplanet migrates through. We take into account three constraints from the solar system and from protoplanetary discs: (1) the masses of the terrestrial planets and the comet reservoirs in Neptune's scattered disc and the Oort cloud are consistent with a primordial planetesimal population of a few Earth masses per AU, (2) evidence from the asteroid belt and the Kuiper belt indicates that the characteristic planetesimal diameter is 100 km, and (3) observations of protoplanetary discs indicate that the dust is stirred by weak turbulence; this gas turbulence also excites the inclinations of planetesimals. Our nominal model built on these constraints results in maximum protoplanet masses of 0.1 Earth masses. Ignoring constraint (1) above, we show that even a planetesimal population of 1,000 Earth masses, corresponding to 50 Earth masses per AU, fails to produce cold gas giants (although it successfully forms hot and warm gas giants). We conclude that a massive planetesimal reservoir is in itself insufficient to produce cold gas giants. The formation of cold gas giants by planetesimal accretion additionally requires that planetesimals are small and that the turbulent stirring is very weak, thereby violating all three above constraints.