Fragment-based drug discovery (FBDD) relies on the premise that the fragment binding mode will be conserved on subsequent expansion to a larger ligand. However, no general condition has been established to explain when fragment binding modes will be conserved. We show that a remarkably simple condition can be developed in terms of how fragments coincide with binding energy hot spots—regions of the protein where interactions with a ligand contribute substantial binding free energy—the locations of which can easily be determined computationally. Because a substantial fraction of the free energy of ligand binding comes from interacting with the residues in the energetically most important hot spot, a ligand moiety that sufficiently overlaps with this region will retain its location even when other parts of the ligand are removed. This hypothesis is supported by eight case studies. The condition helps identify whether a protein is suitable for FBDD, predicts the size of fragments required for screening, and determines whether a fragment hit can be extended into a higher affinity ligand. Our results show that ligand binding sites can usefully be thought of in terms of an anchor site, which is the top-ranked hot spot and dominates the free energy of binding, surrounded by a number of weaker satellite sites that confer improved affinity and selectivity for a particular ligand and that it is the intrinsic binding potential of the protein surface that determines whether it can serve as a robust binding site for a suitably optimized ligand.