Humans can make remarkable social inferences by watching each other's behavior. In many cases, however, people can also make social inferences about agents whose behavior they cannot see, based only on the physical evidence left behind. We hypothesized that this capacity is supported by a form of mental event reconstruction. Under this account, observers derive social inferences by reconstructing the agent's behavior, based on the physical evidence that revealed their presence. We present a computational model of this idea, embedded in a Bayesian framework for action understanding, and show that its predictions match human inferences with high quantitative accuracy. Specifically, Experiment 1 shows that people can infer where an agent came from and which goal they pursued in a room, all from a small pile of cookie crumbs. Experiment 2 shows that people can explicitly reconstruct the actions that the agent took, and these reconstructed trajectories can predict the entry point and goal inferences from Experiment 1. Finally, Experiment 3 shows that people can also infer whether one or two agents were in a room based on the position of two piles of cookie crumbs. Our results shed light on how people extract social information from the physical world.