Neuromorphic high-frequency 3D dancing pose estimation in dynamic environment

Zhang, Zhongyang; Chai, Kaidong; Yu, Haowen; Majaj, Ramzi; Walsh, Francesca; Wang, Edward; Mahbub, Upal; Siegelmann, Hava T.; Kim, Dong-Hyun; Rahman, Tauhidur

doi:10.1016/j.neucom.2023.126388

Cited by 7 publications

(2 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This experiment demonstrated the potential of applying the proposed system in a popular high-level vision problem, human detection, and pose estimation. Event cameras are particularly well suited for detection tasks that involve fast motion and have attracted interest in recent years ( 55 – 57 ). However, previous methods need either the assistance of grayscale images to update the detection ( 55 ) or the initialization of the pose estimation ( 56 ).…”

Section: Resultsmentioning

confidence: 99%

Microsaccade-inspired event camera for robotics

He,

Wang,

Zhou

et al. 2024

Sci. Robot.

View full text Add to dashboard Cite

Neuromorphic vision sensors or event cameras have made the visual perception of extremely low reaction time possible, opening new avenues for high-dynamic robotics applications. These event cameras’ output is dependent on both motion and texture. However, the event camera fails to capture object edges that are parallel to the camera motion. This is a problem intrinsic to the sensor and therefore challenging to solve algorithmically. Human vision deals with perceptual fading using the active mechanism of small involuntary eye movements, the most prominent ones called microsaccades. By moving the eyes constantly and slightly during fixation, microsaccades can substantially maintain texture stability and persistence. Inspired by microsaccades, we designed an event-based perception system capable of simultaneously maintaining low reaction time and stable texture. In this design, a rotating wedge prism was mounted in front of the aperture of an event camera to redirect light and trigger events. The geometrical optics of the rotating wedge prism allows for algorithmic compensation of the additional rotational motion, resulting in a stable texture appearance and high informational output independent of external motion. The hardware device and software solution are integrated into a system, which we call artificial microsaccade–enhanced event camera (AMI-EV). Benchmark comparisons validated the superior data quality of AMI-EV recordings in scenarios where both standard cameras and event cameras fail to deliver. Various real-world experiments demonstrated the potential of the system to facilitate robotics perception both for low-level and high-level vision tasks.

show abstract

Section: Resultsmentioning

confidence: 99%

Microsaccade-inspired event camera for robotics

He,

Wang,

Zhou

et al. 2024

Sci. Robot.

View full text Add to dashboard Cite

show abstract

“…Heatmap-based learning is slower than the direct prediction of coordinates, but it is less sensitive to slight differences that may occur owing to human annotations because it accepts the surroundings of coordinates more generously. The proposed heatmap-based landmark detection model used a Gaussian heatmap generated around landmark coordinates as the ground truth (Figure 2), and the dice coefficient loss (L dice ) and weighted L1 loss (L wl ) were used as the loss functions [11].…”

Section: Learning Of Heatmap-based Landmark Detectionmentioning

confidence: 99%

Deep Learning Method for Precise Landmark Identification and Structural Assessment of Whole-Spine Radiographs

Noh,

Lee,

Bae

et al. 2024

Bioengineering

View full text Add to dashboard Cite

This study measured parameters automatically by marking the point for measuring each parameter on whole-spine radiographs. Between January 2020 and December 2021, 1017 sequential lateral whole-spine radiographs were retrospectively obtained. Of these, 819 and 198 were used for training and testing the performance of the landmark detection model, respectively. To objectively evaluate the program’s performance, 690 whole-spine radiographs from four other institutions were used for external validation. The combined dataset comprised radiographs from 857 female and 850 male patients (average age 42.2 ± 27.3 years; range 20–85 years). The landmark localizer showed the highest accuracy in identifying cervical landmarks (median error 1.5–2.4 mm), followed by lumbosacral landmarks (median error 2.1–3.0 mm). However, thoracic landmarks displayed larger localization errors (median 2.4–4.3 mm), indicating slightly reduced precision compared with the cervical and lumbosacral regions. The agreement between the deep learning model and two experts was good to excellent, with intraclass correlation coefficient values >0.88. The deep learning model also performed well on the external validation set. There were no statistical differences between datasets in all parameters, suggesting that the performance of the artificial intelligence model created was excellent. The proposed automatic alignment analysis system identified anatomical landmarks and positions of the spine with high precision and generated various radiograph imaging parameters that had a good correlation with manual measurements.

show abstract