Real-Time Gestural Control of Robot Manipulator Through Deep Learning Human-Pose Inference

Martin, Jesus Bujalance; Moutarde, Fabien

doi:10.1007/978-3-030-34995-0_51

Cited by 10 publications

(3 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…En la literatura existen pocos trabajos respecto a la adquisición de datos humanos a través de sensores como cámaras para manipuladores no antropomórficos. En (Martin and Moutarde, 2019), se realiza un primer enfoque para resolver este problema, donde se utiliza una cámara RGB-D y OpenPose para el control de un brazo UR3 aplicando un proceso de CI. Otros trabajos como (Gao et al, 2019) presentan una idea similar utilizando BodyPoseNet para la extracción de características del cuerpo para manipulación paralela dual, teniendo en cuenta la posición del brazo contrario.…”

Section: Estado Del Arteunclassified

Tracking humano visual aplicado a manipuladores no antropomórficos para imitación

Prados,

López,

Barber

et al. 2023

XLIV Jornadas De Automática: Libro De Actas: Universidad De Zaragoza, Escuela De Ingeniería Y Arquitectura, 6, 7 Y 8 De Septiem

View full text Add to dashboard Cite

En este trabajo se presenta un algoritmo de detección humana basado en el uso de único sensor RGB-D para demostrar movimientos teniendo en cuenta los puntos clave del brazo humano (hombro-codo-muñeca) para controlar un brazo no antropomórfico. El método desarrollado se basa en la detección de dichos puntos significativos del brazo humano y su mapeo al robot, un proceso de filtrado Gaussiano que permite suavizar los movimientos y el ruido del sensor y un algoritmo de optimización que obtiene la configuración más cercana al brazo humano sin generar colisiones con el entorno ni consigo mismo. La novedad de este método se basa en la utilización de los puntos clave del brazo humano, teniendo en cuenta el efector final y el codo, para obtener la configuración más similar para un brazo no antropomórfico. Se han generado pruebas con diferentes movimientos realizados a distintas velocidades para validar nuestro método, comprobando su eficacia en el efector final del robot.

show abstract

Section: Estado Del Arteunclassified

Tracking humano visual aplicado a manipuladores no antropomórficos para imitación

Prados,

López,

Barber

et al. 2023

XLIV Jornadas De Automática: Libro De Actas: Universidad De Zaragoza, Escuela De Ingeniería Y Arquitectura, 6, 7 Y 8 De Septiem

View full text Add to dashboard Cite

show abstract

“…The research studies in [24], [25] faced issues and took compromises in teleoperating a robot relying on an OpenPose detection system. To overcome jerks and jumps of the robotic counterpart, the authors chose heuristic solutions such as euclidean filters on subjects' links or predetermined operating area of the agent to match the camera Field of View (FoV).…”

Section: ) Human-robot Interaction In Simulationmentioning

confidence: 99%

Open-VICO: An Open-Source Gazebo Toolkit for Vision-based Skeleton Tracking in Human-Robot Collaboration

Fortini¹,

Mattia²,

Gandarias³

et al. 2022

Preprint

View full text Add to dashboard Cite

Simulation tools are essential for robotics research, especially for those domains in which safety is crucial, such as Human-Robot Collaboration (HRC). However, it is challenging to simulate human behaviors, and existing robotics simulators do not integrate functional human models. This work presents Open-VICO , an open-source toolkit to integrate virtual human models in Gazebo focusing on vision-based human tracking. In particular, Open-VICO allows to combine in the same simulation environment realistic human kinematic models, multicamera vision setups, and human-tracking techniques along with numerous robot and sensor models thanks to Gazebo. The possibility to incorporate pre-recorded human skeleton motion with Motion Capture systems broadens the landscape of human performance behavioral analysis within Human-Robot Interaction (HRI) settings. To describe the functionalities and stress the potential of the toolkit four specific examples, chosen among relevant literature challenges in the field, are developed using our simulation utils: i) 3D multi-RGB-D camera calibration in simulation, ii) creation of a synthetic human skeleton tracking dataset based on OpenPose, iii) multi-camera scenario for human skeleton tracking in simulation, and iv) a human-robot interaction example. The key of this work is to create a straightforward pipeline which we hope will motivate research on new vision-based algorithms and methodologies for lightweight human-tracking and flexible human-robot applications.

show abstract

“…This breakthrough has stimulated the skeletal modality interest since it proved to be sufficient to describe and understand the motion of a given action without any background context. This has made pose-based action recognition preferred over other modalities on a huge amount of real-time scenarios for human action recognition such as human-robot interaction [24], [3], medical rehabilitative applications [25], [8] or pedestrian action prediction [12], [11], [13]. Some commonly used learning architectures for pose-based action recognition include 1D/2D convolutional networks [9], [27], recurrent networks [1], [39], a combination of one of the latter with attention mechanisms [21], [16] or Graph-based models [56], [50].…”

Section: A Pose-based Action Recognitionmentioning

confidence: 99%

TrouSPI-Net: Spatio-temporal attention on parallel atrous convolutions and U-GRUs for skeletal pedestrian crossing prediction

Gesnouin

Pechberti

Stanciulescu

et al. 2021

2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)

Self Cite

View full text Add to dashboard Cite

Understanding the behaviors and intentions of pedestrians is still one of the main challenges for vehicle autonomy, as accurate predictions of their intentions can guarantee their safety and driving comfort of vehicles. In this paper, we address pedestrian crossing prediction in urban traffic environments by linking the dynamics of a pedestrian's skeleton to a binary crossing intention. We introduce TrouSPI-Net: a context-free, lightweight, multi-branch predictor. TrouSPI-Net extracts spatio-temporal features for different time resolutions by encoding pseudo-images sequences of skeletal joints' positions and processes them with parallel attention modules and atrous convolutions. The proposed approach is then enhanced by processing features such as relative distances of skeletal joints, bounding box positions, or ego-vehicle speed with U-GRUs. Using the newly proposed evaluation procedures for two large public naturalistic data sets for studying pedestrian behavior in traffic: JAAD and PIE, we evaluate TrouSPI-Net and analyze its performance. Experimental results show that TrouSPI-Net achieved 76% F1 score on JAAD and 80% F1 score on PIE, therefore outperforming current state-of-the-art while being lightweight and context-free.

show abstract

Real-Time Gestural Control of Robot Manipulator Through Deep Learning Human-Pose Inference

Cited by 10 publications

References 9 publications

Tracking humano visual aplicado a manipuladores no antropomórficos para imitación

Tracking humano visual aplicado a manipuladores no antropomórficos para imitación

Open-VICO: An Open-Source Gazebo Toolkit for Vision-based Skeleton Tracking in Human-Robot Collaboration

TrouSPI-Net: Spatio-temporal attention on parallel atrous convolutions and U-GRUs for skeletal pedestrian crossing prediction

Contact Info

Product

Resources

About