Photovoltaics (PV) on building rooftops is a major contributor to the decarbonization of energy systems. We simulate the PV energy yield potential for 2.5 million individual roofs in three German regions. We cumulate the results for each single roof to calculate the cost‐potential curves for the three cities Berlin, Cologne, and Hanover. These curves give the amount of electricity that can be generated at less than a given cost per kWh. We find that these curves have the shape of a hockey stick. Neglecting the dependence of PV investment on building size and thus on the system sizes causes largely different cost‐potential curves that differ by 11%–18% for flat roofs due to their heterogeneous building size distribution. The cost‐potential curves of the three cities are very similar when appropriately normalized, for example, by the local solar irradiation and the settlement area of the city, despite substantial variations in population density. This allows for an extrapolation of our results. For Germany, we reveal an upper limit for the total electricity generation from rooftop PV of 762 TWh/a with cost as low as 6.9 ct/kWh without accounting for area losses due to chimneys, air conditioning systems, and so forth. We estimate the actual potential to be at least half of that figure.