A prerequisite for decreasing the intensification of energy in buildings is to evaluate and understand the influencing factors of building energy performance (BEP). These factors include building envelope features and outdoor climactic conditions, among others. Based on the importance of the influencing factors in the development of the building energy prediction model, various researchers are continuously employing different types of factors based on their popularity in academic literature, without a proper investigation of the most relevant factors, which, in some cases, potentially leads to poor model performance. However, this can be due to the absence of an adequate comprehensive analysis or review of all factors influencing BEP ubiquitously. Therefore, this paper conducts a holistic and comprehensive review of studies that have explored the various factors influencing energy use in residential and commercial buildings. In total, 74 research articles were systematically selected from the Scopus, ScienceDirect, and Institute of Electrical Electronics Engineers (IEEE) databases. Subsequently, by means of a systematic and bibliometric analysis, this paper comprehensively analyzed several important factors influencing BEP. The results reveals the important factors (such as windows and roofs) and engendered or shed light on the application of some energy-efficient strategies such as the utilization of a green roof and photovoltaic (PV) window, among others.