We develop a new approach to extracting the physical consequences of S-duality of four-dimensional N = 4 super Yang-Mills (SYM) and its string theory dual, based on SL(2, Z) spectral theory.We observe that CFT observables O, invariant under SL(2, Z) transformations of a complexified gauge coupling τ , admit a unique spectral decomposition into a basis of square-integrable functions. This formulation has direct implications for the analytic structure of N = 4 SYM data, both perturbatively and non-perturbatively in all parameters. These are especially constraining for the structure of instantons: k-instanton sectors are uniquely determined by the zero-and one-instanton sectors, and Borel summable series around k-instantons have convergence radii with simple k-dependence. In large N limits, we derive the existence and scaling of non-perturbative effects, in both N and the 't Hooft coupling, which we exhibit for certain N = 4 SYM observables. An elegant benchmark for these techniques is the integrated stress tensor multiplet four-point function, conjecturally determined by [1] for all τ for SU (N ) gauge group; we derive and elucidate its form, and explain how the SU (2) case is the simplest possible observable consistent with SL(2, Z)-invariant perturbation theory.These results have ramifications for holography. We explain how O , the ensemble average of O over the N = 4 supersymmetric conformal manifold with respect to the Zamolodchikov measure, is cleanly isolated by the spectral decomposition. We prove that the large N limit of O equals the large N , large 't Hooft coupling limit of O. Holographically speaking, O = Osugra, its value in type IIB supergravity on AdS5 × S 5 . This result, which extends to all orders in 1/N , embeds ensemble averaging into the traditional AdS/CFT paradigm. The statistics of the SL(2, Z) ensemble exhibit both perturbative and non-perturbative 1/N effects. We discuss further implications and generalizations to other AdS compactifications of string/M-theory.