Context. Low-surface-brightness galaxies (LSBs) contribute to a significant fraction of all the galaxies in the Universe. Ultra-diffuse galaxies (UDGs) form a subclass of LSBs that has attracted a lot of attention in recent years (although its definition may vary between studies). Although UDGs are found in large numbers in galaxy clusters, groups, and in the field, their formation and evolution are still very much debated. Aims. Using a comprehensive set of multiwavelength data from the NGVS (optical), VESTIGE (Hα narrowband), and GUViCS (UV) surveys, we studied a sample of 64 diffuse galaxies and UDGs in the Virgo cluster to investigate their formation history. Methods. We analyzed the photometric colors and surface-brightness profiles of these galaxies and then compared them to models of galaxy evolution, including ram-pressure stripping (RPS) events to infer any possible strong interactions with the hot cluster gas in the past. Results. While our sample consists mainly of red LSBs, which is typical in cluster environments, we found evidence of a color variation with the cluster-centric distance. Blue, H i-bearing, star-forming diffuse galaxies are found at larger distances from the cluster center than the rest of the sample. The comparison of our models with multifrequency observations suggests that most of the galaxies of the sample might have undergone a strong RPS event in their lifetime, on average 1.6 Gyr ago (with a large dispersion, and RPS still ongoing for some of them). This process resulted in the transformation of initially gas-rich diffuse blue galaxies into gas-poor and red ones that form the dominant population now, the more extreme UDGs having undergone the process in a more distant past on average. Conclusions. The RPS in dense environments could be one of the major mechanisms for the formation of the large number of quiescent UDGs we observe in galaxy clusters.