EARLY FUSION for Goal Directed Robotic Vision

Walsman, Aaron; Bisk, Yonatan; Gabriel, Saadia; Misra, Dipendra; Artzi, Yoav; Choi, Yejin; Fox, Dieter

doi:10.1109/iros40897.2019.8968165

Cited by 7 publications

(4 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our experiments, we also find that a naive late fusion architecture minus the optical flow yields poor results in RL settings (see Section 5.2). This observation is consistent with recent findings in related domains like visual navigation (Walsman et al, 2019).…”

Section: Introductionsupporting

confidence: 94%

Reinforcement Learning with Latent Flow

Shang,

Wang,

Srinivas

et al. 2021

Preprint

View full text Add to dashboard Cite

Temporal information is essential to learning effective policies with Reinforcement Learning (RL). However, current state-of-the-art RL algorithms either assume that such information is given as part of the state space or, when learning from pixels, use the simple heuristic of frame-stacking to implicitly capture temporal information present in the image observations. This heuristic is in contrast to the current paradigm in video classification architectures, which utilize explicit encodings of temporal information through methods such as optical flow and two-stream architectures to achieve state-of-the-art performance. Inspired by leading video classification architectures, we introduce the Flow of Latents for Reinforcement Learning (Flare), a network architecture for RL that explicitly encodes temporal information through latent vector differences. We show that Flare (i) recovers optimal performance in state-based RL without explicit access to the state velocity, solely with positional state information, (ii) achieves state-ofthe-art performance on pixel-based challenging continuous control tasks within the DeepMind control benchmark suite, namely quadruped walk, hopper hop, finger turn hard, pendulum swing, and walker run, and is the most sample efficient model-free pixel-based RL algorithm, outperforming the prior model-free state-ofthe-art by 1.9× and 1.5× on the 500k and 1M step benchmarks, respectively, and (iv), when augmented over rainbow DQN, outperforms this state-of-the-art level baseline on 5 of 8 challenging Atari games at 100M time step benchmark.

show abstract

Section: Introductionsupporting

confidence: 94%

Reinforcement Learning with Latent Flow

Shang,

Wang,

Srinivas

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Another issue is the investigation of the network architecture for handling an action or language-based instruction beyond an input image. Since the image and another input are used to control the camera agent, designing an effective fusion model is an important problem [20], [42], [43]. An early fusion approach for goal directed navigation fuses the goal information with the input state followed by a convolution process to generate the feature for navigation [42].…”

Section: A Visual Navigation With and Without External Informationmentioning

confidence: 99%

“…Since the image and another input are used to control the camera agent, designing an effective fusion model is an important problem [20], [42], [43]. An early fusion approach for goal directed navigation fuses the goal information with the input state followed by a convolution process to generate the feature for navigation [42]. Alternatively, a gated attention architecture fuses language-based instruction with the input state [43].…”

Section: A Visual Navigation With and Without External Informationmentioning

confidence: 99%

View-Action Representation Learning for Active First-Person Vision

Cavallaro

2021

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

In visual navigation, a moving agent equipped with a camera is traditionally controlled by an input action and the estimation of the features from a sensory state (i.e. the camera view) is treated as a pre-processing step to perform high-level vision tasks. In this paper, we present a representation learning approach that, instead, considers both state and action as inputs. We condition the encoded feature from the state transition network on the action that changes the view of the camera, thus describing the scene more effectively. Specifically, we introduce an action representation module that generates decoded higher dimensional representations from an input action to increase the representational power. We then fuse the output from the action representation module with the intermediate response of the state transition network that predicts the future state. To enhance the discrimination capability among predictions from different input actions, we further introduce triplet ranking loss and N-tuplet loss functions, which in turn can be integrated with the regression loss. We demonstrate the proposed representation learning approach in reinforcement and imitation learning-based mapless navigation tasks, where the camera agent learns to navigate only through the view of the camera and the performed action, without external information.

show abstract

“…Concatenation is the most popular method for performing early fusion, and is also known as additive methods [353]. Examples include the studies of Liu et al [370], Walsman et al [371], Petscharnig et al [372], and Sindagi et al [373]. A multiplication approach takes the product of the properties, as in the work of Zadeh et al [374] for fusing audio and video, or in the study of Fukui et al [375] for fusing visual and text properties through bilinear pooling.…”

Section: Multimodalitymentioning

confidence: 99%

Automated joining element design for high product variety in the manufacturing industry

Eggink

View full text Add to dashboard Cite

Product variety and the manufacturing complexity that it induces are continuously increasing. This poses a challenge in the product development process and, consequently, the design of joints. Joining elements define the manner in which a permanent joint is created between parts. Joining element design is an ambiguous manual task with limited automation solutions. Thus, it can lead to long, iterative, error-prone development trajectories that may result in costly rework. Hence, automation solutions for joining element design must be intelligent. However, simply ensuring the intelligent automation of joining element design is insufficient. Modular design, through the approaches of modularization and commonalization, enables manufacturers to cope with the complexity induced by product variety. Unfortunately, modular design approaches have not yet considered joining elements. Hence, this dissertation study sought to answer the following research question: "How can joining element design be automated for high-variety products?"This dissertation presents a framework for automating joining element design. The framework is structured into smaller design problems, which will guide designers through the process of designing joining elements. This structuring will also enable designers to evaluate and assess artificial intelligence (AI) for each design problem. Moreover, the design problems will enable designers to identify unused AI techniques, such as machine learning.These techniques were conceptualized and several were implemented for validation in this study. A market-validated database with automotive Body-in-White structures was used. The validation included the use of decision trees to predict joining technologies and the number of joining elements. Both prediction tasks seem promising to implement in early design phases due to their simplicity.Next, this study validated two approaches for predicting joining locations. The first was a straight-forward evolutionary algorithm for distributing spot welds over contact regions. However, this algorithm's performance fell off quickly for nontrivial design problems with high solution spaces.The second approach was to use convolutional neural networks to predict spot weld locations, which provided robust and promising results. This study first explored a voxel-based regression and classification task by drawing locations in a grid-like structure. Classification with a segmentation approach produced more robust results due to the spatial dependencies of classes. Supervised machine learning enabled the consideration of knowledge of successful designs in new design problems. This concept was subsequently enriched with nongeometric data using a branding approach. However, these multimodal machine learning models did not improve the joining locations. Furthermore, the models could not extract the i ii

show abstract

EARLY FUSION for Goal Directed Robotic Vision

Cited by 7 publications

References 24 publications

Reinforcement Learning with Latent Flow

Reinforcement Learning with Latent Flow

View-Action Representation Learning for Active First-Person Vision

Automated joining element design for high product variety in the manufacturing industry

Contact Info

Product

Resources

About