Order-Aware Generative Modeling Using the 3D-Craft Dataset

Chen, Zhuoyuan; Srinet, Kavya; Qi, Charles R.; Fan, Haoqi; Ma, Jerry; Zitnick, Larry; Guo, Demi; Xiao, Tong; Xie, Sihong; Chen, Xinlei; Szlam, Arthur; Tulsiani, Shubham; Yu, Haonan; Gray, Jonathan

doi:10.1109/iccv.2019.00185

Cited by 6 publications

(5 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We compute two metrics: the Chamfer distance between the ground truth final shape and the shape output by each model. We also compute a normalized Mistakes to Complete (MTC) score, proposed in [7]. MTC computes the average percentage of steps where a model gives wrong poses predictions.…”

Section: Setupmentioning

confidence: 99%

See 1 more Smart Citation

Translating a Visual LEGO Manual to a Machine-Executable Plan

Wang¹,

Zhang²,

Mao³

et al. 2022

Preprint

View full text Add to dashboard Cite

We study the problem of translating an image-based, stepby-step assembly manual created by human designers into machineinterpretable instructions. We formulate this problem as a sequential prediction task: at each step, our model reads the manual, locates the components to be added to the current shape, and infers their 3D poses. This task poses the challenge of establishing a 2D-3D correspondence between the manual image and the real 3D object, and 3D pose estimation for unseen 3D objects, since a new component to be added in a step can be an object built from previous steps. To address these two challenges, we present a novel learning-based framework, the Manual-to-Executable-Plan Network (MEPNet), which reconstructs the assembly steps from a sequence of manual images. The key idea is to integrate neural 2D keypoint detection modules and 2D-3D projection algorithms for highprecision prediction and strong generalization to unseen components. The MEPNet outperforms existing methods on three newly collected LEGO manual datasets and a Minecraft house dataset.

show abstract

Section: Setupmentioning

confidence: 99%

“…As a community, we would like to build machines that can assist humans in constructing and assembling complex objects, such as block worlds [7], LEGO models [9], and furniture [35]. The assembly task involves a sequence of actions that move different 3D parts to desired poses.…”

Section: Introductionmentioning

confidence: 99%

Translating a Visual LEGO Manual to a Machine-Executable Plan

Wang¹,

Zhang²,

Mao³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Label-Conditioned Generative Models. To generate label-conditioned block placements, we adapt the VoxelCNN model presented in Chen et al [12]. Given a 3D patch of a scene with block type information and a global view with occupancy information, VoxelCNN predicts next block type and placement.…”

Section: Abstructions For Build() Operationsmentioning

confidence: 99%

“…Our fine-tuning runs for an additional 4 epochs and for each class we select the model with the best performance on the validation set. We use the same train-val split as Chen et al [12], but save 50% of the validation set as our test set. Averaging across categories, we achieve a top-10 accuracy of 66.0% and average 7.50 consecutively correct blocks.…”

Section: Abstructions For Build() Operationsmentioning

confidence: 99%

See 1 more Smart Citation

Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning

Burns,

Manning,

Fei-Fei

2021

Preprint

View full text Add to dashboard Cite

Although virtual agents are increasingly situated in environments where natural language is the most effective mode of interaction with humans, these exchanges are rarely used as an opportunity for learning. Leveraging language interactions effectively requires addressing limitations in the two most common approaches to language grounding: semantic parsers built on top of fixed object categories are precise but inflexible and end-to-end models are maximally expressive, but fickle and opaque. Our goal is to develop a system that balances the strengths of each approach so that users can teach agents new instructions that generalize broadly from a single example. We introduce the idea of neural abstructions: a set of constraints on the inference procedure of a label-conditioned generative model that can affect the meaning of the label in context. Starting from a core programming language that operates over abstructions, users can define increasingly complex mappings from natural language to actions. We show that with this method a user population is able to build a semantic parser for an open-ended house modification task in Minecraft. The semantic parser that results is both flexible and expressive: the percentage of utterances sourced from redefinitions increases steadily over the course of 191 total exchanges, achieving a final value of 28%.

show abstract

Learning 3D-Craft Generation with Predictive Action Neural Network

Liu

Zuo

et al. 2021

MultiMedia Modeling

View full text Add to dashboard Cite

Order-Aware Generative Modeling Using the 3D-Craft Dataset

Cited by 6 publications

References 15 publications

Translating a Visual LEGO Manual to a Machine-Executable Plan

Translating a Visual LEGO Manual to a Machine-Executable Plan

Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning

Learning 3D-Craft Generation with Predictive Action Neural Network

Contact Info

Product

Resources

About