More than 1000 respondents in Sweden (2013) and the US (2014) were asked to report their subjective opinions and attitudes about situations that caused them regret, concern, worry, and anxiety. US respondents self-identified as Black. Although exploratory factor analyses extracted many latent factors from the 80 questions, a common latent inner factor was extracted from five questions that examined key psychological phenomena: worry at the present time, bothersome concerns in the present, regret for the past, anxiety about the future, and unpleasant experience in the past. Confirmatory factor analyses and structural equation modeling of the latent variables (SEM/LV) provided convincing evidence of the existence of a common latent inner factor in both countries. Because each of the five key phenomena reflected concerns involving the self, the common latent inner factor was labeled "Being unable to detach from concerns involving the self." Then, the same latent inner factor was also confirmed in SEM/LV of combinations of data from Swedish and US Black respondents, and from respondents in a previous study (Japanese, and US respondents who identified as White; Hayase, 2016). Women, younger people, and people with lower levels of education were less able to detach from concerns involving the self than men, older people, and people with higher levels of education. Confirmatory factor analyses by SEM/LV provided additional evidence of the existence of a common latent inner factor for the five phenomena, worry, bothersome concerns, regret, anxiety, and unpleasant experience. Psychological and philosophical implications of the latent inner factor with regard to genuine happiness were discussed.