Spatial patterns of genetic relatedness among contemporary samples reflect the past movements of their ancestors. Our ability to untangle this spatial history has the potential to improve dramatically given that we can now infer the ultimate description of genetic relatedness, an ancestral recombination graph (ARG). By extending spatial methods previously applied to trees, we generalize a model of Brownian dispersal to ARGs, thereby accounting for correlations along a chromosome when computing the likelihood-based estimates of dispersal rate and locations of genetic ancestors. We develop an efficient algorithm that allows us to apply our method to complex ARGs, scalable to thousands of samples. We evaluate our method’s ability to reconstruct spatial histories using simulations. Surprisingly, despite using the fullest information available in the data, we find that our dispersal estimates are biased, highlighting a discrepancy between the histories of recombinant lineages and Brownian dispersal models. We identify potential resolutions to this problem based on relaxing the constraints that ARGs place on the movement of lineages and show that ARG-based spatial inference can be used to effectively track the geographic history of admixed individuals. Approaches like this will be key to understanding the interplay of migration, recombination, drift, and adaptation in geographically spread populations.