Alleviating the Burden of Labeling: Sentence Generation by Attention Branch Encoder–Decoder Network

Target-dependent UNITER: A Transformer-Based Multimodal Language Comprehension Model for Domestic Service Robots

“…A representative work on the latter strand is [14], which describes a comprehension module trained by human-generated representations for use as a "critic" of the referring expression generation, and the re-evaluation of the generated representations. ABEN [15] is an encoder-decoder model that generates instructions about objects in an image. This model performs the processing in the opposite direction to our method.…”

“…In the field of VRE, datasets using real images include RefCLEF [22], RefCOCO [22], and GuessWhat [23], while those using synthetic images include CLEVR-Ref+ [24]. Public datasets for MLU-FI include PFN-PIC [21] and WRS-PV [15], both of which consist of images and fetching instructions for objects in the images. The PFN-PIC dataset contains real images with objects scattered in four boxes, whereas the WRS-PV dataset contains images collected in the standard simulator of World Robot Summit Partner Robot Challenge [25] by DSRs moving around the room.…”

Case Relation Transformer: A Crossmodal Language Generation Model for Fetching Instructions

“…Indeed, the quality of generated sentences for the FIG tasks was far worse than that for simple image captioning tasks [2]. In fact, there is a big gap in the quality between reference sentences and generated sentences by conventional methods, as shown in Section V. Although the FIG task was handled in previous studies [2], existing models did not handle the target object and destination simultaneously [2]. Therefore, they could not generate instruction sentences, such as "Move the blue flip flop to the lower left box.," where the target object is "the blue flip flop," and the destination is "the lower left box."…”

“…The FIG task is challenging because ambiguity in a sentence depends not only on the target object but also on surrounding objects. Indeed, the quality of generated sentences for the FIG tasks was far worse than that for simple image captioning tasks [2]. In fact, there is a big gap in the quality between reference sentences and generated sentences by conventional methods, as shown in Section V. Although the FIG task was handled in previous studies [2], existing models did not handle the target object and destination simultaneously [2].…”

“…The CRT consists of three main modules: Case Relation Block (CRB), Transformer encoder, and Transformer decoder. Unlike existing methods [2], the CRT uses the Transformer to integrate the visual and geometry features of objects appearing in the FIG task. Moreover, unlike the Object Relation Transformer (ORT [4]), the CRT can handle both the target object and destination.…”