We present a direct, adaptive solver for the Poisson equation which can achieve any prescribed order of accuracy. It is based on a domain decomposition approach using local spectral approximation, as well as potential theory and the fast multipole method. In two space dimensions, the algorithm requires O(NK) work where N is the number of discretization points and K is the desired order of accuracy.