Arginine vasopressin (AVP) and related peptides have diverse effects on social behaviors in vertebrates, sometimes promoting affiliative interactions and sometimes aggressive or antisocial responses. The type of influence, in at least some species, depends on social contexts, including the sex of the individuals in the interaction and/or on the levels of peptide within brain circuits that control the behaviors. To determine if AVP promotes different responses to same- and other-sex faces in men, and if those effects are dose dependent, we measured the effects of two doses of AVP on subjective ratings of male and female faces. We also tested if any influences persist beyond the time of drug delivery. When AVP was administered intranasally on an initial test day, 20 IU was associated with decreased social assessments relative to placebo and 40 IU, and some of the effects persisted beyond the initial drug delivery and appeared to generalize to novel faces on subsequent test days. In single men, those influences were most pronounced, but not exclusive, for male faces, whereas in coupled men they were primarily associated with responses to female faces. Similar influences were not observed if AVP was delivered after placebo on a second test day. In a preliminary analysis, the differences in social assessments observed between men who received 20 and 40 IU, which we suggest primarily reflect lowered social assessments induced by the lower dose, appeared most pronounced in subjects who carry what has been identified as a risk allele for the V1a receptor gene. Together, these results suggest that AVP’s effects on face processing, and possibly other social responses, differ according to dose, depend on relationship status, and may be more prolonged than previously recognized.