Many interpersonal cognitions and behaviors, such as impression formation, communication, and cooperation, are controlled and performed while surrounded by the built environment. This study reviewed over 20 years of literature on the influence of visual environmental factors, particularly brightness, color, and depth, on interpersonal cognition and behavior. Twelve articles on brightness were extracted, showing that the effects of brightness varied significantly depending on individual characteristics, interpersonal relationships, and interactional situations. Thirteen articles on color revealed that various factors such as age, gender, and interpersonal relationships are involved in the influence process. Eight articles that examined influences from depth (spaciousness and height) indicated that influences from depth are a highly complex process. Finally, the study has shown areas for further research, challenges to be overcome, and prospects.