Why do some members of Congress vote more on the extremes of their party than others? I argue that lawmakers representing more homogeneously white districts have greater electoral incentive to moderate their voting records, since the two parties compete more for support of white voters than for the support of minority voters. I provide evidence using roll-call votes from the U.S. House and Senate. I find members representing more homogeneously white districts have more moderate voting records, a finding that holds for Democrats and Republicans. I explore two potential mechanisms: legislator responsiveness and electoral punishment. While legislators do not seem to adjust their voting behavior in response to short-term changes in district racial composition, more homogeneously white districts are found to assess larger vote share penalties on more extreme candidates in general elections. The findings have implications for our understanding of race, representation, and electoral accountability.