Two groups are virtually isomorphic if they can be obtained one from the other via a finite number of steps, where each step consists in taking a finite extension or a finite index subgroup (or viceversa). Virtually isomorphic groups are always quasi-isometric, and a group Γ is quasi-isometrically rigid if every group quasi-isometric to Γ is virtually isomorphic to Γ. In this survey we describe quasi-isometric rigidity results for fundamental groups of manifolds which can be decomposed into geometric pieces. After stating by now classical results on lattices in semisimple Lie groups, we focus on the class of fundamental groups of 3-manifolds, and describe the behaviour of quasi-isometries with respect to the Milnor-Kneser prime decomposition (following Papasoglu and Whyte) and with respect to the JSJ decomposition (following Kapovich and Leeb). We also discuss quasi-isometric rigidity results for fundamental groups of higher dimensional graph manifolds, that were recently defined by Lafont, Sisto and the author. Our main tools are the study of geometric group actions and quasi-actions on Riemannian manifolds and on trees of spaces, via the analysis of the induced actions on asymptotic cones.