We study stability of metric approximations of countable groups with respect to groups endowed with ultrametrics, the main case study being a p-adic analogue of Ulam stability, where we take GL n (Z p ) as approximating groups instead of U(n). For finitely presented groups, the ultrametric nature implies equivalence of the pointwise and uniform stability problems, and the profinite one implies that the corresponding approximation property is equivalent to residual finiteness. Moreover, a group is uniformly stable if and only if its largest residually finite quotient is. We provide several examples of uniformly stable groups: these include finite groups, virtually free groups, some groups acting on rooted trees, and certain lamplighter and (Generalized) Baumslag-Solitar groups.