The D-instanton corrected hypermultiplet moduli space of type II string theory compactified on a Calabi-Yau threefold is known in the type IIA picture to be determined in terms of the generalized Donaldson-Thomas invariants, through a twistorial construction. At the same time, in the mirror type IIB picture, and in the limit where only D3-D1-D(-1)-instanton corrections are retained, it should carry an isometric action of the S-duality group SL(2, Z). We prove that this is the case in the one-instanton approximation, by constructing a holomorphic action of SL(2, Z) on the linearized twistor space. Using the modular invariance of the D4-D2-D0 black hole partition function, we show that the standard Darboux coordinates in twistor space have modular anomalies controlled by period integrals of a Siegel-Narain theta series, which can be canceled by a contact transformation generated by a holomorphic mock theta series.