This article develops W. E. B. Du Bois’s notion of democratic despotism to illustrate the entanglement of popular sovereignty and empire through an excessive form of western self-determination and theorizes how features of this formation remain today. Democratic despotism implies that, in western democracies at the turn of the twentieth century, popular sovereignty was an impulse to partake of the wealth and resources obtained by empire. Western democracies issued a claim to determine themselves (democratically) and others (despotically), in what I call “self-and-other-determination.” I frame the question of imperial democracy within the literature on empire and racial capitalism and the writings of Du Bois, Frantz Fanon, and Saidiya Hartman to conceptualize how racial affective attachments allow citizens past and present to restrict democratic concern to a limited community, whose wealth relies on the imperial exploitation of racialized others. I discuss the absence of these questions in the literature on self-determination and reflect on the implications of my framework for the contemporary rise of right-wing populism.