The detection of object movement that is contingent on one’s own actions (i.e., movements with action contingency) influences social perception of the object; such interactive objects tend to create a good impression. However, it remains unclear whether neural representation of action contingency is associated with subsequent socio-cognitive evaluation of “contacting agents”, or whether the appearance of agents (e.g., face- or non-face-like avatars) is essential for this effect. In this study, we conducted a functional magnetic resonance imaging (fMRI) task with two phases: contact (contact with face- or non-face-like avatars moving contingently or non-contingently) and recognition (rating a static image of each avatar). Deactivation of the frontoparietal self-agency network and activation of the reward network were the main effects of action contingency during the contact phase, consistent with previous findings. During the recognition phase, static avatars that had previously moved in a contingent manner deactivated the frontal component of the frontoparietal network (bilateral insula and inferior-middle frontal gyri), regardless of person-like appearance. Our results imply that frontal deactivation may underlie the effect of action contingency on subsequent social perception, independent of person-like appearance.