We performed imaging and manipulation of epitaxially grown Pb adatom structures on the Si(100) surface by non-contact atomic force microscopy (NC-AFM) in ultra-high-vacuum (UHV) and at cryogenic temperatures. We observe several distinct contrast modes during imaging, which we assign to termination of the scanning probe tip by either a single Si, or single Pb, atom, via quantitative comparison of atomic resolution force spectroscopy experiments with ab initio density functional theory (DFT) simulations. We show that the Pb adatom structures can be controllably manipulated via mechano-chemical means, and identify a novel semi-deterministic manipulation strategy that arises from the combination of low temperature operation and the asymmetric diffusion barriers present on the Si(100) surface.