Grain growth is generally driven to minimize the overall grain boundary energy. However, for low-angle grain boundaries the requirement that lattice planes be continuous across the boundary gives rise to a coupling between the normal motion of the grain boundary and the tangential motion of the lattice. We show through phase-field crystal simulations this coupling in polycrystalline systems can give rise to a rigid body translation of the lattice as a grain shrinks. The process is mediated by significant climb of the dislocations in the boundary and dislocation reactions at the trijunctions. Thus the grain growth process is coupled to vacancy diffusion processes as well as the dynamics of grain trijunctions. Moreover, grain shrinkage can cease because of dislocation behavior near the trijunction, illustrating that this coupling can have an influence on the grain growth process in polycrystals.