It is known that the group of all diffeomorphisms of a manifold determines uniquely the topological and smooth structure of the manifold itself. We specify a possibly large class of diffeomorphism groups which satisfy this property. In particular, so does the group of contact diffeomorphisms.