In this article, we propose new gradient recovery schemes for the BEM-based Finite Element Method (BEM-based FEM) and Virtual Element Method (VEM). Supporting general polytopal meshes, the BEM-based FEM and VEM are highly flexible and efficient tools for the numerical solution of boundary value problems in two and three dimensions. We construct the recovered gradient from the gradient of the finite element approximation via local averaging. For the BEM-based FEM, we show that, under certain requirements on the mesh, superconvergence of the recovered gradient is achieved, which means that it converges to the true gradient at a higher rate than the untreated gradient. Moreover, we propose a simple and very efficient a posteriori error estimator, which measures the difference between the unprocessed and recovered gradient as an error indicator. Since the BEM-based FEM and VEM are specifically suited for adaptive refinement, the resulting adaptive algorithms perform very well in numerical examples.