Energy recovery ventilators (ERVs) are used to recover sensible and latent heat to reduce the heating and cooling load caused by outdoor air intake into indoor environments. The efficiency of the heat exchanger, comprising flow channels and constituent materials, determines the heat exchange performance of the ERV. Understanding the heat and mass transfer mechanisms in the ERV and optimizing the geometry and materials of the flow channel are essential for improving the ERV performance. Herein, numerical methods for predicting and clarifying the hygrothermal transfer mechanism in the heat exchange element of an ERV were developed, and their prediction accuracy was confirmed by analysing the exchange efficiencies in the scaled-down ERV unit model. The verified numerical model was applied to sensitivity analyses to clarify the rate-limiting factors influencing the total heat exchange efficiency of the ERV. Our findings have clarified that under Japan Industrial Standard cooling conditions, a 50-fold increase in the thermal conductivity of the spacer plate, the total heat recovery performance was enhanced by 13%, as for a 2-fold increase in the moisture conductivity, the performance was enhanced by 20%. The findings of this research can be expected to contribute to the energy saving effect in buildings.