This study examined the determinants of social withdrawal using data from a survey by the Tokyo Metropolitan Government Office for Youth Affairs and Public Safety (2008). In addition, this study identified young people who showed an affinity for social withdrawal although they were not in a state of withdrawal, and examined the determinants of an affinity for social withdrawal. The results of stepwise discriminant analysis showed that factors such as social phobia, depression, violence, and emotional bonds with family differentiated between the general youth group and the social withdrawal group and the "affinity group". Social phobia, violence, and refusal to be interfered in self-decision making differentiated between the social withdrawal group and the "affinity group". This study shows that an "affinity group" should be cared as well as an actual withdrawal group.