In recent years, virtual reality (VR) is at its maturity level for real practical exercises amongst many fields of studies, especially in the virtual walkthrough exploration system of cultural heritage (CH). However, this study remains scattered and limited. This work presents a systematic review that maps out the usability and accessibility issues that are challenging in using VR in CH. We identified 45 challenges that are mapped into five problem groups: system design, development process, technology, assessment process and knowledge transfer. This mapping is then used to propose 58 recommendations to improve the usability and accessibility of VR in CH that are categorized in three different recommendation groups namely, discovery and planning, design and development, and finally the assessment factors. This analysis identified the persistence in certain accessibility and usability problems such as there is a limit in navigating the view and space that constraint the users’ free movement and the navigation control is not ideal with the keyboard arrow button. This work is important because it provides an overview of usability and accessibility based challenges that are faced in applying, developing, deploying and assessing VR in the usage of digitalizing CH and proposed a great number of constructive recommendations to guide future studies. The main contribution of this paper is the mapping of usability and accessibility challenges into categories and the development of recommendations based on the identified problems.