This paper is concerned with the development of a fast spectral method for solving direct and indirect boundary integral equations in 3D-potential theory. Based on a Galerkin approximation and the Fast Fourier Transform, the proposed method is a generalization of the precorrected-FFT technique to handle not only single-layer potentials but also double-layer potentials and higher-order basis functions. Numerical examples utilizing piecewise linear shape functions are presented to illustrate the performance of the method.