In this paper, authors systematically selected and reviewed articles related to stereoscopic displays and their advances, with a special focus on perception, interaction, and corresponding challenges. The aim was to understand interaction‐related problems, provide possible explanations, and identify factors that limit their applications. Despite promising advancements, there are still issues that researchers in the field fail to explain precisely. The two major problems in stereoscopic viewing are, compared with the real world, objects are perceived to be smaller than they actually are and there are discomfort and visual syndromes. Furthermore, there is general agreement that humans underestimate their egocentric distance in a virtual environment (VE). Our analysis revealed that in the real world, distance estimation is about 94% accurate, but in VE, it is only about 80% accurate. This problem could reduce the efficacy of different sensory motor‐based applications where interaction is important. Experts from human factors, computing, psychology, and others have studied contributing factors such as types of perception/response method, quality of graphics, associated stereoscopic conditions, experience in virtual reality (VR), and distance signals. This paper discusses the factors requiring further investigation if the VR interaction is to be seamlessly realized. In addition, engineering research directions aiming at improving current interaction performances are recommended.