RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder

Xiao, Shitao; Liu, Zheng; Shao, Yingxia; Cao, Zhao

doi:10.18653/v1/2022.emnlp-main.35

Cited by 25 publications

(18 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We call this VideoDex-BC-Open. We ablate the type of visual representation and prior use by trying an initialization using the VGG16 network (Simonyan and Zisserman 2014) (VideoDex-VGG) and the MVP network (He et al, 2022; Xiao et al, 2022) (VideoDex-MVP) based representation trained for robot learning. We ablate the need for a two-stream policy, instead training a single NDP for both hand and wrist.…”

Section: Resultsmentioning

confidence: 99%

“…We compared using our approach with MVP (VideoDex-MVP) (Xiao et al, 2022) and VGG (VideoDex-VGG) (Simonyan and Zisserman 2014) and their performance was below VideoDex using Nair et al (2022). This is likely because both encoders are much larger than the ResNet18 (He et al, 2015) we use and require a lot more training time than feasible on human videos.…”

Section: Resultsmentioning

confidence: 99%

“…This is likely because both encoders are much larger than the ResNet18 (He et al, 2015) we use and require a lot more training time than feasible on human videos. However, VideoDex-MVP still performs better than VideoDex-VGG, which indicates that using a visual prior trained on human data does in fact help, as Xiao et al (2022) trained the representation in self-supervised fashion on videos and use the embeddings to perform robotics tasks in simulation. We see in Table 6, that while visual priors are important, action priors are more impactful.…”

Section: Resultsmentioning

confidence: 99%

“…This insight of leveraging human videos to aid robotics is not new and has seen immense attention from the vision community at large (Damen et al, 2018; Goyal et al, 2017; Grauman et al, 2022). However, most of this prior work tends to use human data as a mechanism for pretraining just the visual representation (Nair et al 2018, 2022; Pinto et al, 2016; Sermanet et al, 2018; Xiao et al, 2022), much like how deep learning has been used as a pretraining tool in related areas of computer vision (Chen et al, 2020; He et al, 2017) and natural language processing (Brown et al, 2020; Devlin et al, 2018).…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Learning dexterity from human hand motion in internet videos

Shaw,

Bahl,

Sivakumar

et al. 2024

The International Journal of Robotics Research

View full text Add to dashboard Cite

To build general robotic agents that can operate in many environments, it is often useful for robots to collect experience in the real world. However, unguided experience collection is often not feasible due to safety, time, and hardware restrictions. We thus propose leveraging the next best thing as real world experience: videos of humans using their hands. To utilize these videos, we develop a method that retargets any 1st person or 3rd person video of human hands and arms into the robot hand and arm trajectories. While retargeting is a difficult problem, our key insight is to rely on only internet human hand video to train it. We use this method to present results in two areas: First, we build a system that enables any human to control a robot hand and arm, simply by demonstrating motions with their own hand. The robot observes the human operator via a single RGB camera and imitates their actions in real-time. This enables the robot to collect real-world experience safely using supervision. See these results at https://robotic-telekinesis.github.io . Second, we retarget in-the-wild human internet video into task-conditioned pseudo-robot trajectories to use as artificial robot experience. This learning algorithm leverages action priors from human hand actions, visual features from the images, and physical priors from dynamical systems to pretrain typical human behavior for a particular robot task. We show that by leveraging internet human hand experience, we need fewer robot demonstrations compared to many other methods. See these results at https://video-dex.github.io

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Learning dexterity from human hand motion in internet videos

Shaw,

Bahl,

Sivakumar

et al. 2024

The International Journal of Robotics Research

View full text Add to dashboard Cite

show abstract

“…However, a strong decoder may negatively impact sequence representation quality (Lu et al 2021). Recent approaches such as SimLM (Wang et al 2023) and RetroMAE (Xiao et al 2022) address this issue by adopting shallow decoders with limited past context access and enhanced decoding mechanisms.…”

Section: Related Workmentioning

confidence: 99%

Some Like It Small: Czech Semantic Embedding Models for Industry Applications

Bednář,

Náplava,

Barančíková

et al. 2024

AAAI

View full text Add to dashboard Cite

This article focuses on the development and evaluation of Small-sized Czech sentence embedding models. Small models are important components for real-time industry applications in resource-constrained environments. Given the limited availability of labeled Czech data, alternative approaches, including pre-training, knowledge distillation, and unsupervised contrastive fine-tuning, are investigated. Comprehensive intrinsic and extrinsic analyses are conducted, showcasing the competitive performance of our models compared to significantly larger counterparts, with approximately 8 times smaller size and 5 times faster speed than conventional Base-sized models. To promote cooperation and reproducibility, both the models and the evaluation pipeline are made publicly accessible. Ultimately, this article presents practical applications of the developed sentence embedding models in Seznam.cz, the Czech search engine. These models have effectively replaced previous counterparts, enhancing the overall search experience for instance, in organic search, featured snippets, and image search. This transition has yielded improved performance.

show abstract

Foundation models for bioinformatics

Chen,

Wei,

Gao

2024

Quant. Biol.

View full text Add to dashboard Cite

Transformer‐based foundation models such as ChatGPTs have revolutionized our daily life and affected many fields including bioinformatics. In this perspective, we first discuss about the direct application of textual foundation models on bioinformatics tasks, focusing on how to make the most out of canonical large language models and mitigate their inherent flaws. Meanwhile, we go through the transformer‐based, bioinformatics‐tailored foundation models for both sequence and non‐sequence data. In particular, we envision the further development directions as well as challenges for bioinformatics foundation models.

show abstract

RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder

Cited by 25 publications

References 0 publications

Learning dexterity from human hand motion in internet videos

Learning dexterity from human hand motion in internet videos

Some Like It Small: Czech Semantic Embedding Models for Industry Applications

Foundation models for bioinformatics

Contact Info

Product

Resources

About