We show that for any lattice Veech group in the mapping class group Mod.S/ of a closed surface S, the associated 1 S -extension group is a hierarchically hyperbolic group. As a consequence, we prove that any such extension group is quasi-isometrically rigid.