I consider the forms of control, which went “untreated” by 1970s “revisionist” penality literature (in other words, I wonder whether the categories of human beings who are (mostly) not found in prisons have something in common). I take as starting point that the “temporary slavery” which is the punishment of imprisonment, emerged historically as related to the “free” condition of those punished. Forms of control instead for the “unfree” are not to be included in “(penal) imprisonment” and could be understood as “domestic” forms of control expressed, originally, in the idea of “Pater Familias.” This form of control is not punishment but is a permanent condition deemed appropriate for given categories of human beings, such as “children,” “women,” “slaves,” and what I call “the mad and other non-persons.” I first examine how imprisonment (as punishment) emerged, after the end of servitude in Europe, as a sort of “memory of slavery,” to enforce a principle of subordination dedicated to “the free.” Then, I look at the mechanisms of social control for those who are not socially perceived as “free.” Finally, I attempt at sketching the process of expansion of mechanisms of subordination—for the free and the unfree—beyond European borders.