After the accident at the Fukushima Daiichi Nuclear Power Station, food products from the areas affected by the accident suffered reputational damage worldwide. Therefore, the present study aimed to identify the factors associated with people refraining from purchasing foods produced in affected areas to avoid radioactive materials. The study also aimed to clarify the time trends for the avoidance of foods produced in Fukushima Prefecture. We used data from “A survey on consumer awareness of reputational damages” conducted by the Japanese Consumers Customer Agency and implemented statistical analysis. Even if the year since the accident differed, “living with children”, “knowing detailed information about food inspections”, and “not being able to accept radiation risk even if the level is below the standard” were commonly associated. Not only did this study reveal that some people’s risk perceptions are fixed even when new knowledge is provided, but it also suggests that the implementation of food inspection can promote reputational damage. Additionally, the avoidance of Fukushima food products was found to decrease as time passed after the Fukushima accident. The results of this study may help develop countermeasures against reputational damage to food products after future nuclear disasters.