VNect

Abstract

Fig. 1. We recover the full global 3D skeleton pose in real-time from a single RGB camera, even wireless capture is possible by streaming from a smartphone (left). It enables applications such as controlling a game character, embodied VR, sport motion analysis and reconstruction of community video (right). Community videos (CC BY) courtesy of Real Madrid C.F. [2016] and RUSFENCING-TV [2017].We present the first real-time method to capture the full global 3D skeletal pose of a human in a stable, temporally consistent manner using a single RGB camera. Our method combines a new convolutional neural network (CNN) based pose regressor with kinematic skeleton fitting. Our novel fullyconvolutional pose formulation regresses 2D and 3D joint positions jointly in real time and does not require tightly cropped input frames. A real-time kinematic skeleton fitting method uses the CNN output to yield temporally stable 3D global pose reconstructions on the basis of a coherent kinematic skeleton. This makes our approach the first monocular RGB method usable in real-time applications such as 3D character control-thus far, the only monocular methods for such applications employed specialized RGB-D cameras. Our method's accuracy is quantitatively on par with the best offline 3D monocular RGB pose estimation methods. Our results are qualitatively comparable to, and sometimes better than, results from monocular RGB-D approaches, such as the Kinect. However, we show that our approach is more broadly applicable than RGB-D solutions, i.e., it works for outdoor scenes, community videos, and low quality commodity RGB cameras.

StarMap for Category-Agnostic Keypoint and Viewpoint Estimation

“…The results in Table. 10 show that our DepthMap representation achieves very close performance with the original design of Zhou et al [51], while saving about 1/5 network parameter (from the depth-regression sub-network). We also compare with Mehta et al [20], who also use a map representation for 3D coordinates. Instead of directly using the (u, v) coordinate from 2D heatmap (with a weak-perspective camera model), they regress the full (x, y, z) coordinates at the peak heatmap location with a full-perspective camera model.…”

Beyond reach: Do symmetric changes in motor costs affect decision making? A registered report

“…Particularly, experimental studies in the field of neuroeconomics and consumer behavior employ increasingly more ecologically valid paradigms (Hui et al, 2009;Rangel et al, 2008;Sanfey et al, 2006;Sharp et al, 2012); the current study suggests that finegrained recording of full-body movements in the laboratory can allow to track the time course of the underlying cognitive processes. Furthermore, the current work provides basis for principally new field studies tracking decision-makers' behavior in the real world: recent progress in computer vision (e.g., Toshev & Szegedy, 2014;Cao et al, 2017;Mehta et al, 2017) enables the researchers to reconstruct walking trajectories during decision making from video stream data, eliminating the need for dedicated motion capture hardware. These trajectories can then be used for reverse inference of the walkers' cognitive processes across domains.…”

Deformation Capture via Soft and Stretchable Sensor Arrays

“…However, these methods only capture 2D skeletal information. Predicting 3D poses directly from 2D RGB images has been demonstrated using offline methods [Bogo et al 2016;Tekin et al 2016;Zhou et al 2016] and in online settings [Mehta et al 2017]. Monocular depth cameras provide additional information and have been shown to aid robust skeletal tracking [Ganapathi et al 2012;Ma and Wu 2014;Shotton et al 2013;Taylor et al , 2012 and enable dense surface reconstruction even under deformation [Dou et al 2016;Newcombe et al 2015;Zollhöfer et al 2014].…”