The relationship between Japan and Southeast Asian countries has shifted from occupation to a better political and economic partnership in the last few decades. However, in Penang, Malaysia, bitter legacies rooted in the occupation period are continuously reproduced through the educational system, folklore and the existence of the Penang War Museum. This article aims to identify the role of direct encounters between the host community in Penang and the Japanese visitors in shaping a better image of the Japanese. Employing qualitative methods, findings from a series of observations and in-depth interviews in Penang reveal several interesting facts. First collective memories inherited from the Japanese colonization still exist although the economic and political relationship with Japan has reached its best level. Second, the role of direct interaction completed by the economic benefit from tourism is effective in breaking the cultural block between groups inheriting conflict in the past. Moreover, direct interaction between Japanese visitors and the host community in Penang has significantly produced a better mutual understanding, compared to the perception of people who did not have any interaction with the Japanese visitors. It shows that tourism can be considered as the alternative soft diplomacy to complete the puzzle of public diplomacy through its people-to-people interaction.