We construct finite-and infinite-dimensional non-relativistic extensions of the Newton-Hooke and Carroll (A)dS algebras using the algebra expansion method, starting from the (anti-)de Sitter relativistic algebra in D dimensions. These algebras are also shown to be embedded in different affine Kac-Moody algebras. In the three-dimensional case, we construct Chern-Simons actions invariant under these symmetries. This leads to a sequence of non-relativistic gravity theories, where the simplest examples correspond to extended Newton-Hooke and extended (post-)Newtonian gravity together with their Carrollian counterparts.