The memories of “comfort women” have been discursively represented primarily within the dynamic discussions that define narratives regarding the universal human rights of women. In this article, I attempt to re-read the history of the comfort women system against the paradigmatic human rights narratives of gender-based violence while contextualizing its history within the legacy of colonial racism. The understanding of the comfort women system as colonial violence that I put forward thus challenges the basic assumptions of current debates on contested memories in East Asia. For this, I draw on haunting aspects of Zainichi Korean documentary filmmaker Soonam Park’s archivization of human suffering during the colonial period and on Michel Foucault’s notion of biopolitics. I further discuss the contrasting implications of W. E. B. Du Bois’s visits to Japan and to the ruins of the Warsaw Ghetto in light of Michael Rothberg’s reading of Du Bois and of his conceptual reflections on “marking caesuras.” In this regard, I characterize Imperial Japan as the absent empire of postcolonial grammatology, underscoring thereby the manner in which Eurocentrism, binary thinking, and habitual orientation around the color line inhabit and determine much of the work carried out under the banner of postcolonial studies. “Enter the ghost, exit the ghost, re-enter the ghost.” – Hamlet.