We study properties of self-duality symmetry in the Cardy-Rabinovici model. The Cardy-Rabinovici model is the 4d U (1) gauge theory with electric and magnetic matters, and it enjoys the SL(2, Z) self-duality at low-energies. SL(2, Z) self-duality does not realize in a naive way, but we notice that the ST p duality transformation becomes the legitimate duality operation by performing the gauging of Z N 1-form symmetry with including the level-p discrete topological term. Due to such complications in its realization, the fusion rule of duality defects becomes a non-group-like structure, and thus the self-duality symmetry is realized as a non-invertible symmetry. Moreover, for some fixed points of the self-duality, the duality symmetry turns out to have a mixed gravitational anomaly detected on a K3 surface, and we can rule out the trivially gapped phase as a consequence of anomaly matching. We also uncover how the conjectured phase diagram of the Cardy-Rabinovici model satisfies this new anomaly matching condition.