The current polarization of elites in the U.S., particularly in Congress, is frequently ascribed to the emergence of cohorts of ideologically extreme legislators replacing moderate ones. Politicians, however, do not operate as isolated agents, driven solely by their preferences. They act within organized parties, whose leaders exert control over the rank-and-file, directing support for and against policies. This paper shows that the omission of party discipline as a driver of political polarization is consequential for our understanding of this phenomenon. We present a multi-dimensional voting model and identification strategy designed to decouple the ideological preferences of lawmakers from the control exerted by their party leadership. Applying this structural framework to the U.S. Congress between 1927-2018, we find that the influence of leaders over their rank-and-file has been a growing driver of polarization in voting, particularly since the 1970s. In 2018, party discipline accounts for around 65% of the polarization in roll call voting. Our findings qualify the interpretation of -and in two important cases subvert -a number of empirical claims in the literature that measures polarization with models that lack a formal role for parties.