3D Semantic Label Transfer in Human-Robot Collaboration

Rozenberszki, Dávid; Sörös, Gábor; Szeier, Szilvia; Lörincz, András

doi:10.1109/iccvw54120.2021.00294

Cited by 5 publications

(11 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…During navigation, the software selects distances, directions and salient features for verbal communication. The method has promise for real-life scenarios since similar dense semantic models of the space can be learned, see, e.g., [13] and the references therein.…”

Section: Indoor Navigation Instruction Generationmentioning

confidence: 99%

“…Verbal communication about the environment requires semantic and instance maps that can work in real time, and shared object semantics between the human and the guiding system. We review the components developed previously in [13].…”

Section: Our Frameworkmentioning

confidence: 99%

“…Maps of this type may take the form of a 3D mesh of the surfaces of the objects in the environment with semantically labeled mesh points. There are other forms, e.g., a voxel map of the environment with semantically labeled voxel points, see, e.g., [13,34]. Since 3D body position and pose can be estimated (see Section 2.4) and the allocentric map can be translated and rotated, the robot can talk about the environment in terms of the egocentric coordinates of the partner.…”

Section: Semantic and Instance Map Modulementioning

confidence: 99%

“…The semantic back-projection process labels independent voxels within the environment with the appropriate categories retrieved from the 2D RGB frames and depth information. We generate the 3D semantic and instance maps from different viewpoints in real-time [13], e.g., from the viewpoint of the human partner that can simplify machine instructions, e.g., the machine may talk about to your left/right and so on.…”

Section: Semantic and Instance Map Modulementioning

confidence: 99%

“…Upon estimating the position and the pose of the partner, the map enables the machine to approximate what the human partner may see, i.e., to see 'through the eyes' of the partner, and to communicate according to the partner's egocentric coordinate system. This technology component needs shared object semantics as considered in [13]. In addition to egocentric frames of reference, instance maps also enable object-centric communication, and we use both: we say 'to your right', or 'to the left of the chair'.…”

Section: Semantic and Instance Mapmentioning

confidence: 99%

See 4 more Smart Citations

AI Technologies for Machine Supervision and Help in a Rehabilitation Scenario

Baranyi

Melício

Gaál

et al. 2022

MTI

View full text Add to dashboard Cite

We consider, evaluate, and develop methods for home rehabilitation scenarios. We show the required modules for this scenario. Due to the large number of modules, the framework falls into the category of Composite AI. Our work is based on collected videos with high-quality execution and samples of typical errors. They are augmented by sample dialogues about the exercise to be executed and the assumed errors. We study and discuss body pose estimation technology, dialogue systems of different kinds and the emerging constraints of verbal communication. We demonstrate that the optimization of the camera and the body pose allows high-precision recording and requires the following components: (1) optimization needs a 3D representation of the environment, (2) a navigation dialogue to guide the patient to the optimal pose, (3) semantic and instance maps are necessary for verbal instructions about the navigation. We put forth different communication methods, from video-based presentation to chit-chat-like dialogues through rule-based methods. We discuss the methods for different aspects of the challenges that can improve the performance of the individual components. Due to the emerging solutions, we claim that the range of applications will drastically grow in the very near future.

show abstract

Section: Indoor Navigation Instruction Generationmentioning

confidence: 99%

Section: Our Frameworkmentioning

confidence: 99%

Section: Semantic and Instance Map Modulementioning

confidence: 99%

Section: Semantic and Instance Map Modulementioning

confidence: 99%

Section: Semantic and Instance Mapmentioning

confidence: 99%

See 3 more Smart Citations