Second-generation exoplanet imagers using extreme adaptive optics (ExAO) and coronagraphy have demonstrated their great potential for studying close circumstellar environments and for detecting new companions and helping to understand their physical properties. However, at very small angular separation, their performance in contrast is limited by several factors: diffraction by the complex telescope pupil (central obscuration and spiders) not perfectly canceled by the coronagraph, residual dynamic wavefront errors, chromatic wavefront errors, and wavefront errors resulting from noncommon path aberrations (NCPAs). These latter are differential aberrations between the visible wavefront sensing path of the ExAO system and the near-infrared science path in which the coronagraph is located. In a previous work, we demonstrated the use of a Zernike wavefront sensor called ZELDA for sensing NCPAs in the VLT/SPHERE exoplanet imager and their compensation with the high-order deformable mirror of the instrument. These early tests on the internal light source led to encouraging results for the attenuation of the quasi-static speckles at very small separation. In the present work, we move to the next step with the on-sky validation of NCPA compensation with ZELDA. With an improved procedure for the compensation of NCPAs, we start by reproducing previous results on the internal source. We show that the amount of aberration integrated between 1 and 15 cycles/pupil (c/p) is decreased by a factor of approximately five, which translates into a gain in raw contrast of between 2 and 3 at separations below 300 mas. On sky, we demonstrate that NCPA compensation works in closed loop, leading to an attenuation of the amount of aberration by a factor of approximately two. However, we identify a loss of sensitivity for the sensor that is only partly explained by the difference in Strehl ratio between the internal and on-sky measurements. Our simulations show that the impact of ExAO residuals on ZELDA measurements is negligible for integration times beyond a few tenths of a second. Coronagraphic imaging on sky is improved in raw contrast by a factor of 2.5 at most in the ExAO-corrected region. We use coronagraphic image reconstruction based on a detailed model of the instrument to demonstrate that both internal and on-sky raw contrasts can be precisely explained, and we establish that the observed performance after NCPA compensation is no longer limited by an improper compensation for aberration but by the current apodized-pupil Lyot coronagraph design. We finally conclude that a coronagraph upgrade combined to a proper NCPA compensation scheme could easily bring a gain in raw contrast of a factor of two to three below 200 mas.