Photon upconversion is promising for many applications. However, the potential of lanthanide doped upconverter materials is typically limited by low absorption coefficients and low upconversion quantum yields (UCQY) under practical irradiance of the excitation. Modifying the photonic environment can strongly enhance the spontaneous emission and therefore also the upconversion luminescence. Additionally, the non-linear nature of the upconversion processes can be exploited by an increased local optical field introduced by photonic or plasmonic structures. In combination, both processes may lead to a strong enhancement of the UCQY at simultaneously lower incident irradiances. Here, we use a comprehensive 3D computation-based approach to investigate how absorption, upconversion luminescence, and UCQY of an upconverter are altered in the vicinity of spherical gold nanoparticles (GNPs). We use Mie theory and electrodynamic theory to compute the properties of GNPs. The parameters obtained in these calculations were used as input parameters in a rate equation model of the upconverter β-NaYF4: 20% Er3+. We consider different diameters of the GNP and determine the behavior of the system as a function of the incident irradiance. Whether the UCQY is increased or actually decreased depends heavily on the position of the upconverter in respect to the GNP. Whereas the upconversion luminescence enhancement reaches a maximum around a distance of 35 nm to the surface of the GNP, we observe strong quenching of the UCQY for distances <40 nm and a UCQY maximum around 125 to 150 nm, in the case of a 300 nm diameter GNP. Hence, the upconverter material needs to be placed at different positions, depending on whether absorption, upconversion luminescence, or UCQY should be maximized. At the optimum position, we determine a maximum UCQY enhancement of 117% for a 300 nm diameter GNP at a low incident irradiance of 0.01 W/cm2. As the irradiance increases, the maximum UCQY enhancement decreases to 20% at 1 W/cm2. However, this UCQY enhancement translates into a significant improvement of the UCQY from 12.0% to 14.4% absolute.