How do people hold others responsible? Responsibility judgments are affected not only by what actually happened, but also by what could have happened if things had turned out differently. Here, we look at how replaceability -- the ease with which a person could have been replaced by someone else -- affects responsibility. We develop the counterfactual replacement model which runs simulations of alternative scenarios to determine the probability that the outcome would have been different if the person of interest had been replaced. The model predicts that a person is held more responsible when it would have been more difficult to replace them. To test the model's predictions, we design a paradigm that quantitatively varies replaceability by manipulating the number of replacements as well as the probability with which each replacement would have been available. Across three experiments featuring increasingly complex scenarios, we show that the model explains participants' responsibility judgments well in both social and physical settings, and better than alternative models that rely only on features of what actually happened.