This study examined how different forms of childhood family victimization are associated with the attitudinal (not actual action) refusal of wife abuse among women and men in rural Bangladesh. It included 1,929 randomly selected married women and men. Of the sample, 31.3% (Men = 49.3%, Women = 13.5%) attitudinally refused overall wife abuse, 38.5% (Men = 53.2%, Women = 23.8%) refused emotional abuse, 67.0% (Men = 82.5%, Women = 51.6%) refused physical abuse, 78.0% (Men = 88.6%, Women = 67.4%) refused abuse on wife’s disobeying family obligations, and 32.3% (Men = 50.3%, Women = 14.6%) refused abuse on challenging male authority. Multivariate logistic regression revealed that the odds ratio (ORs) of the attitudinal refusal of overall wife abuse were 1.75 (p = .041) for the childhood non-victims of emotional abuse and 2.31 (p < .001) for the victims of mild emotional abuse, compared to the victims of severe emotional abuse. On the other hand, the ORs of the overall refusal of abuse were 1.84 (p = .031) for the non-victims of physical abuse and 1.29 (p = .465) for the victims of mild physical abuse, compared to the childhood victims of severe physical abuse. Data further revealed that the childhood non-victimization of physical abuse increased all types of attitudinal refusal of wife abuse, e.g., emotional abuse, physical abuse, abuse on disobeying family obligations, and abuse on challenging male authority. Compared to the childhood experiences of severe emotional abuse, data also indicated that childhood exposure to mild emotional abuse might increase the attitudinal refusal of wife abuse on a few issues, e.g., abuse on disobeying family obligations, abuse on challenging male authority, and physical abuse. It appeared that childhood experiences of family victimization greatly influence different types of attitudinal refusal of wife abuse. We argue that the issue of childhood victimization should be brought to the forefront in the discourse. We recommend that state machinery and social welfare agencies should expend significant efforts to stop child abuse within the family and in other areas of society in rural Bangladesh.