Static and Animated 3D Scene Generation from Free-form Text Descriptions

Huq, Faria; Ahemad, Nafees; Iqbal, Anindya

doi:10.48550/arxiv.2010.01549

Cited by 1 publication

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For increasing the complexity, SA3D [247] introduced the proof of concept for 3D scene generation from text, which is different from previous works on 3D scene generation as it allows free-form text descriptions. Therefore, they showed a two-stage pipeline that can generate static and animated scenes using a transformer-based text encoder with a multi-head decoder for predicting object-specific features per head to create an abstract layout.…”

Section: Manipulationmentioning

confidence: 99%

“…More importantly, research in [120,247] showed that T2Vo for 2D can be extended to 3D, which is far more challenging due to the perception of an additional variable. Among many practical applications of this research, field analysis such as car accidents [306], military tactical planning [402], house design [122], and movie creation [259] are possible.…”

Section: Visual Tasksmentioning

confidence: 99%

“…First attempts in this domain used retrieval models [306,307] that are primarily limited to a fixed set of rules. However, recent progress in the retrieval task using deep learning models of transformer [247] and CNN [243,244] improved this to free-form text for video creation. Although the retrieval task with deep learning models is fascinating, the restriction of specified visual output restrains its practical use for real-world applications.…”

Section: Generative Modelsmentioning

confidence: 99%

See 2 more Smart Citations

A Review of Multi-Modal Learning from the Text-Guided Visual Processing Viewpoint

Ullah

Lee

et al. 2022

Sensors

View full text Add to dashboard Cite

For decades, co-relating different data domains to attain the maximum potential of machines has driven research, especially in neural networks. Similarly, text and visual data (images and videos) are two distinct data domains with extensive research in the past. Recently, using natural language to process 2D or 3D images and videos with the immense power of neural nets has witnessed a promising future. Despite the diverse range of remarkable work in this field, notably in the past few years, rapid improvements have also solved future challenges for researchers. Moreover, the connection between these two domains is mainly subjected to GAN, thus limiting the horizons of this field. This review analyzes Text-to-Image (T2I) synthesis as a broader picture, Text-guided Visual-output (T2Vo), with the primary goal being to highlight the gaps by proposing a more comprehensive taxonomy. We broadly categorize text-guided visual output into three main divisions and meaningful subdivisions by critically examining an extensive body of literature from top-tier computer vision venues and closely related fields, such as machine learning and human–computer interaction, aiming at state-of-the-art models with a comparative analysis. This study successively follows previous surveys on T2I, adding value by analogously evaluating the diverse range of existing methods, including different generative models, several types of visual output, critical examination of various approaches, and highlighting the shortcomings, suggesting the future direction of research.

show abstract

Section: Manipulationmentioning

confidence: 99%