Solid undeformable particles surrounded by a liquid medium or interface may propel themselves by altering their local environment. Such nonmechanical swimming is at work in autophoretic swimmers, whose selfgenerated field gradient induces a slip velocity on their surface, and in interfacial swimmers, which exploit unbalance in surface tension. In both classes of systems, swimmers with intrinsic asymmetry have received the most attention but self-propulsion is also possible for particles that are perfectly isotropic. The underlying symmetry-breaking instability has been established theoretically for autophoretic systems but has yet to be observed experimentally for solid particles. For interfacial swimmers, several experimental works point to such a mechanism, but its understanding has remained incomplete. The goal of this work is to fill this gap. Building on an earlier proposal, we first develop a point-source model that may be applied generically to interfacial or phoretic swimmers. Using this approximate but unifying picture, we show that they operate in very different regimes and obtain analytical predictions for the propulsion velocity and its dependence on swimmer size and asymmetry. Next, we present experiments on interfacial camphor disks showing that they indeed self-propel in an advection-dominated regime where intrinsic asymmetry is irrelevant and that the swimming velocity increases sublinearly with size. Finally, we discuss the merits and limitations of the point-source model in light of the experiments and point out its broader relevance.