Climate sensitivity is a key metric used to assess the magnitude of global warming given increased CO2 concentrations. The geological past can provide insights into climate sensitivity; however, on timescales of millions of years, factors other than CO2 can drive climate, including paleogeographic forcing and solar luminosity. Here, through an ensemble of climate model simulations covering the period 150–35 million years ago, we show that climate sensitivity to CO2 doubling varies between ∼3.5 and 5.5 °C through this time. These variations can be explained as a nonlinear response to solar luminosity, evolving surface albedo due to changes in ocean area, and changes in ocean circulation. The work shows that the modern climate sensitivity is relatively low in the context of the geological record, as a result of relatively weak feedbacks due to a relatively low CO2 baseline, and the presence of ice and relatively small ocean area in the modern continental configuration.