Ivory's Lemma is a geometrical statement in the heart of J. Ivory's calculation of the gravitational potential of a homeoidal shell. In the simplest planar case, it claims that the diagonals of a curvilinear quadrilateral made by arcs of confocal ellipses and hyperbolas are equal.In the first part of this paper, we deduce Ivory's Lemma and its numerous generalizations from complete integrability of billiards on conics and quadrics. In the second part, we study analogs of Ivory's Lemma in Liouville and Stäckel metrics. Our main focus is on the results of the German school of differential geometry obtained in the late 19 -early 20th centuries that might be lesser know today.In the third part, we generalize Newton's, Laplace's, and Ivory's theorems on gravitational and Coulomb potential of spheres and ellipsoids to the spherical and hyperbolic spaces. V. Arnold extended the results of Newton, Laplace, and Ivory to algebraic hypersurfaces in Euclidean space; we generalize Arnold's theorem to the spaces of constant curvature.