Motivated by recent experiments, we present here an ab initio study of the impact of the phonon transport on the thermal conductance and thermoelectric figure of merit of C60-based single-molecule junctions. To be precise, we combine density functional theory with nonequilibrium Green's function techniques to compute these two quantities in junctions with either a C60 monomer or a C60 dimer connected to gold electrodes, taking into account the contributions of both electrons and phonons. Our results show that for C60 monomer junctions phonon transport plays a minor role in the thermal conductance and, in turn, in the figure of merit, which can reach values on the order of 0.1, depending on the contact geometry. In C60 dimer junctions, phonons are transported less efficiently, but they completely dominate the thermal conductance and reduce the figure of merit as compared to monomer junctions. Thus, claims that by stacking C60 molecules one could achieve high thermoelectric performance, which have been made without considering the phonon contribution, are not justified. Moreover, we analyze the relevance of near-field thermal radiation for the figure of merit of these junctions within the framework of fluctuational electrodynamics. We conclude that photon tunneling can be another detrimental factor for the thermoelectric performance, which has been overlooked so far in the field of molecular electronics. Our study illustrates the crucial roles that phonon transport and photon tunneling can play, when critically assessing the performance of molecular junctions as potential nanoscale thermoelectric devices.