The existence of polycrystalline shells has been widely reported in the synthesis of hollow nanoparticles; however, the exact role displayed by the grain boundaries on the stability has been scarcely studied. By including them, in this work, we study for the first time the contribution of the polycrystalline structure in the stability of this unique kind of nanostructures, addressing at the same time, a more realistic modeling of hollow nanoparticles. The role of the polycrystalline structure was studied in gold hollow nanoparticles using molecular dynamics simulations for a wide range of shell thickness and grain sizes. One of the main findings is that the shell thickness necessary for transition from a spherical to a shrunk structure is related to the grain size reduction. The results suggest that to achieve larger hollow nanoparticles, less defective shells are necessary, with single-crystal shells establishing an upper limit in the size that a structure can attain. The cavity shrinkage in a polycrystalline HNP is due to a complex combination of grain diffusion, rotations, dislocation emission, and twining, all of them activated from the grain boundary regions. Our findings suggest that the polycrystalline structure is a crucial parameter to control and improve the stability of the hollow nanoparticles.