Diffusion Probabilistic Modeling for Video Generation

Yang, Ruihan; Srivastava, Prakhar; Mandt, Stephan

doi:10.3390/e25101469

Cited by 53 publications

(5 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Models based on generative adversarial networks (GANs) [42][43][44] and variational autoencoders (VAEs) [45,46] have achieved excellent results in synthetic image generation. Nonetheless, there have recently been significant advancements in static image generation [47,48] and video generation [27,[49][50][51] using diffusion models, even exceeding the performance of GAN and VAE models [27,52].…”

Section: Methodsmentioning

confidence: 99%

Improving Generation and Evaluation of Long Image Sequences for Embryo Development Prediction

Celard,

Seara Vieira,

Sorribes-Fdez

et al. 2024

Electronics

View full text Add to dashboard Cite

Generating synthetic time series data, such as videos, presents a formidable challenge as complexity increases when it is necessary to maintain a specific distribution of shown stages. One such case is embryonic development, where prediction and categorization are crucial for anticipating future outcomes. To address this challenge, we propose a Siamese architecture based on diffusion models to generate predictive long-duration embryonic development videos and an evaluation method to select the most realistic video in a non-supervised manner. We validated this model using standard metrics, such as Fréchet inception distance (FID), Fréchet video distance (FVD), structural similarity (SSIM), peak signal-to-noise ratio (PSNR), and mean squared error (MSE). The proposed model generates videos of up to 197 frames with a size of 128×128, considering real input images. Regarding the quality of the videos, all results showed improvements over the default model (FID = 129.18, FVD = 802.46, SSIM = 0.39, PSNR = 28.63, and MSE = 97.46). On the coherence of the stages, a global stage mean squared error of 9.00 was achieved versus the results of 13.31 and 59.3 for the default methods. The proposed technique produces more accurate videos and successfully removes cases that display sudden movements or changes.

show abstract

Section: Methodsmentioning

confidence: 99%

Improving Generation and Evaluation of Long Image Sequences for Embryo Development Prediction

Celard,

Seara Vieira,

Sorribes-Fdez

et al. 2024

Electronics

View full text Add to dashboard Cite

show abstract

“…The state-of-the-art works of GAN-based approaches were examined by Tyagi and Yadav [23], Frolov et al [1], Zhou et al [24], and Tan et al [25]. On the other hand, on the diffusion models field, with multiple works [20], [21], [26], reviewing the progress of diffusion models in all fields, some articles explore deeply into particular areas, including audio diffusion models [27], diffusion models for video generation [28], diffusion model in vision [29], and text-to-image diffusion models [30], providing a thorough overview of the diffusion model field, as well as an in-depth look at its applications, limitations, and promising future possibilities. However, our work distinctively integrates the latest advancements in both GANs and diffusion models, providing a holistic view of the field.…”

Section: Related Surveys and Study Contributionmentioning

confidence: 99%

Text-to-Image Synthesis With Generative Models: Methods, Datasets, Performance Metrics, Challenges, and Future Direction

Alhabeeb,

Al-Shargabi

2024

IEEE Access

View full text Add to dashboard Cite

Text-to-image synthesis, the process of turning words into images, opens up a world of creative possibilities, and meets the growing need for engaging visual experiences in a world that is becoming more image-based. As machine learning capabilities expanded, the area progressed from simple tools and systems to robust deep learning models that can automatically generate realistic images from textual inputs. Modern, large-scale text-to-image generation models have made significant progress in this direction, producing diversified and high-quality images from text description prompts. Although several methods exist, Generative Adversarial Networks (GANs) have long held a position of prominence. However, diffusion models have recently emerged, with results much beyond those achieved by GANs. This study offers a concise overview of text-to-image generative models by examining the existing body of literature and provide a deeper understanding of this topic. This will be accomplished by providing a concise summary of the development of text-to-image synthesis, previous tools and systems employed in this field, key types of generative models, as well as an exploration of the relevant research conducted on GANs and diffusion models. Additionally, the study provides an overview of common datasets utilized for training the text-toimage model, compare the evaluation metrics used for evaluating the models and addresses the challenges encountered in the field. Finally, concluding remarks are provided to summarize the findings and implications of the study and open issues for further research.

show abstract

“…With its strong generative performance, the diffusion model is the new SOTA among current deep generative models. Works such as [ 152 – 154 ] demonstrated that the diffusion model has a strong ability to understand complex scenarios, and the video diffusion model can generate higher-quality videos. Works such as [ 155 , 156 ] utilized the diffusion model to generate complex and diverse driving scenarios.…”

Section: Prediction Of Autonomous Driving Based On World Modelsmentioning

confidence: 99%

Prospective Role of Foundation Models in Advancing Autonomous Vehicles

Wu,

Gao,

Gao

et al. 2024

Research

View full text Add to dashboard Cite

With the development of artificial intelligence and breakthroughs in deep learning, large-scale foundation models (FMs), such as generative pre-trained transformer (GPT), Sora, etc., have achieved remarkable results in many fields including natural language processing and computer vision. The application of FMs in autonomous driving holds considerable promise. For example, they can contribute to enhancing scene understanding and reasoning. By pre-training on rich linguistic and visual data, FMs can understand and interpret various elements in a driving scene, and provide cognitive reasoning to give linguistic and action instructions for driving decisions and planning. Furthermore, FMs can augment data based on the understanding of driving scenarios to provide feasible scenes of those rare occurrences in the long tail distribution that are unlikely to be encountered during routine driving and data collection. The enhancement can subsequently lead to improvement in the accuracy and reliability of autonomous driving systems. Another testament to the potential of FMs’ applications lies in world models, exemplified by the DREAMER series, which showcases the ability to comprehend physical laws and dynamics. Learning from massive data under the paradigm of self-supervised learning, world models can generate unseen yet plausible driving environments, facilitating the enhancement in the prediction of road users’ behaviors and the off-line training of driving strategies. In this paper, we synthesize the applications and future trends of FMs in autonomous driving. By utilizing the powerful capabilities of FMs, we strive to tackle the potential issues stemming from the long-tail distribution in autonomous driving, consequently advancing overall safety in this domain.

show abstract

Diffusion Probabilistic Modeling for Video Generation

Cited by 53 publications

References 66 publications

Improving Generation and Evaluation of Long Image Sequences for Embryo Development Prediction

Improving Generation and Evaluation of Long Image Sequences for Embryo Development Prediction

Text-to-Image Synthesis With Generative Models: Methods, Datasets, Performance Metrics, Challenges, and Future Direction

Prospective Role of Foundation Models in Advancing Autonomous Vehicles

Contact Info

Product

Resources

About