Reward for altruism and punishment for selfishness are crucial components for the maintenance of society. Past studies have provided strong evidence that people are willing to incur costs to punish selfish behaviors and to reward altruistic behaviors, but how their willingness to do so depends on their relationship with the individuals conducting the anti-social or pro-social behaviors is much less explored. To probe into this question, we devised a three-stage experiment that combined a revised dictator game and third-party reward or punishment. We employed two payoff frameworks, alignment and conflict, and analyzed how third-party’s willingness to reward and punish differed when their interests were either aligned or in conflict with the first-party under observation. We found that due to considerations for personal interests, third-party’s reward and punishment levels deviated from what was deemed “legitimate” by society, that is, the level of reward and punishment that enhances society’s intrinsic motivations to comply with social norms and act pro-socially. When an anti-social behavior was observed, third-party punished less severely under the alignment framework than under the conflict framework; when a pro-social behavior was observed, third-party demonstrated self-serving reward under the alignment framework, but they rewarded altruistically under the conflict framework. These findings provided evidence for third-party’s self-serving reward and punishment.