We present a fast and spectrally accurate numerical scheme for the evaluation of the gyroaveraged electrostatic potential in non-periodic gyrokinetic-Poisson simulations. Our method relies on a reformulation of the gyrokinetic-Poisson system in which the gyroaverage in Poisson's equation is computed for the compactly supported charge density instead of the non-periodic, non-compactly supported potential itself. We calculate this gyroaverage with a combination of two Fourier transforms and a Hankel transform, which has the near optimal run-time complexity O(N ρ (P +P) log(P +P)), where P is the number of spatial grid points,P the number of grid points in Fourier space and N ρ the number of grid points in velocity space. We present numerical examples illustrating the performance of our code and demonstrating geometric convergence of the error.