Augmented reality (AR) and virtual reality (VR) are emerging as a next-generation display platform, surpassing conventional flat panel displays. The AR/VR technologies are garnering considerable attention from both industry and academia as they provide immersive user experiences and smooth interaction between the real and digital worlds. The current AR/VR displays, however, still face several challenges including limited field of view, small eye-box, conflict between the vergence and accommodation, bulky form factor, and heavy weight. This paper aims to provide a comprehensive view of optical elements of AR/VR displays, emphasizing how they enhance the performance and user experience of the AR/VR displays. Various techniques are explained in the following two principal aspects: AR/VR displays based on (1) holographic optical elements and metasurfaces, and (2) combined usage of optical elements. In each section, the characteristics and principles of the optical components are briefly explained. Then, their recent applications addressing the limitations of the conventional AR/VR displays are introduced.