Learning Modality-Specific and -Agnostic Representations for Asynchronous Multimodal Language Sequences

Yang, Dingkang; Kuang, Haopeng; Huang, Shuai; Zhang, Lihua

doi:10.1145/3503161.3547755

Cited by 40 publications

(6 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Using an explicit representation of meshes, ARA‐PReg [HHS*21] designs a spectral ARAP regularization for graph shape generators, constraining them to produce a series of shapes that move as rigidly as possible [SA07]. Recent advances include GenCorres [YHS*24], which brings the ARAPReg‐like principle to implicit neural fields, enforcing cycle consistency and rigidity constraints, and the work of Tang et al [TMW*22], which further extends the modeling of deformation spaces by allowing generation conditioned on user‐given constraints via dragging handles, exposing more control over deformation. HyperDiffusion [EMS*23] can sample from a 4D shape distribution by training a diffusion network to generate MLP weights of an SDF neural field.…”

Section: State‐of‐the‐art Methodsmentioning

confidence: 99%

Recent Trends in 3D Reconstruction of General Non‐Rigid Scenes

Yunus,

Lenssen,

Niemeyer

et al. 2024

Computer Graphics Forum

View full text Add to dashboard Cite

Reconstructing models of the real world, including 3D geometry, appearance, and motion of real scenes, is essential for computer graphics and computer vision. It enables the synthesizing of photorealistic novel views, useful for the movie industry and AR/VR applications. It also facilitates the content creation necessary in computer games and AR/VR by avoiding laborious manual design processes. Further, such models are fundamental for intelligent computing systems that need to interpret real‐world scenes and actions to act and interact safely with the human world. Notably, the world surrounding us is dynamic, and reconstructing models of dynamic, non‐rigidly moving scenes is a severely underconstrained and challenging problem. This state‐of‐the‐art report (STAR) offers the reader a comprehensive summary of state‐of‐the‐art techniques with monocular and multi‐view inputs such as data from RGB and RGB‐D sensors, among others, conveying an understanding of different approaches, their potential applications, and promising further research directions. The report covers 3D reconstruction of general non‐rigid scenes and further addresses the techniques for scene decomposition, editing and controlling, and generalizable and generative modeling. More specifically, we first review the common and fundamental concepts necessary to understand and navigate the field and then discuss the state‐of‐the‐art techniques by reviewing recent approaches that use traditional and machine‐learning‐based neural representations, including a discussion on the newly enabled applications. The STAR is concluded with a discussion of the remaining limitations and open challenges.

show abstract

Section: State‐of‐the‐art Methodsmentioning

confidence: 99%

Recent Trends in 3D Reconstruction of General Non‐Rigid Scenes

Yunus,

Lenssen,

Niemeyer

et al. 2024

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

“…AugmNet , inspired by recent advancements in augmentation literature ( 53 ), is a learning-based approach designed to learn augmentation patterns within the latent space. Unlike traditional methods that apply image augmentations (e.g., rotation, cropping) directly in the pixel space, AugmNet generically implements transformations on the embeddings.…”

Section: Methodsmentioning

confidence: 99%

Multimodal automatic assessment of acute pain through facial videos and heart rate signals utilizing transformer-based architectures

Gkikas,

Tachos,

Andreadis

et al. 2024

Front. Pain Res.

View full text Add to dashboard Cite

Accurate and objective pain evaluation is crucial in developing effective pain management protocols, aiming to alleviate distress and prevent patients from experiencing decreased functionality. A multimodal automatic assessment framework for acute pain utilizing video and heart rate signals is introduced in this study. The proposed framework comprises four pivotal modules: the Spatial Module, responsible for extracting embeddings from videos; the Heart Rate Encoder, tasked with mapping heart rate signals into a higher dimensional space; the AugmNet, designed to create learning-based augmentations in the latent space; and the Temporal Module, which utilizes the extracted video and heart rate embeddings for the final assessment. The Spatial-Module undergoes pre-training on a two-stage strategy: first, with a face recognition objective learning universal facial features, and second, with an emotion recognition objective in a multitask learning approach, enabling the extraction of high-quality embeddings for the automatic pain assessment. Experiments with the facial videos and heart rate extracted from electrocardiograms of the BioVid database, along with a direct comparison to 29 studies, demonstrate state-of-the-art performances in unimodal and multimodal settings, maintaining high efficiency. Within the multimodal context, 82.74% and 39.77% accuracy were achieved for the binary and multi-level pain classification task, respectively, utilizing 9.62 million parameters for the entire framework.

show abstract

“…Deep neural networks are gradually developing toward largescale models [1][2][3][4][5][6]. The changes have brought about an impressive technological breakthrough [7][8][9][10][11][12][13][14][15][16], but applying these technologies to mobile devices such as mobile phones, driverless cars, and tiny robots is difficult. Besides, the source data cannot be obtained in many cases due to data security, such as fingerprints, faces, and medical records images.…”

Section: Introductionmentioning

confidence: 99%

Explicit and Implicit Knowledge Distillation via Unlabeled Data

Wang¹,

Ge²,

Chen³

et al. 2023

Preprint

View full text Add to dashboard Cite

Data-free knowledge distillation is a challenging model lightweight task for scenarios in which the original dataset is not available. Previous methods require a lot of extra computational costs to update one or more generators and their naive imitate-learning lead to lower distillation efficiency. Based on these observations, we first propose an efficient unlabeled sample selection method to replace high computational generators and focus on improving the training efficiency of the selected samples. Then, a class-dropping mechanism is designed to suppress the label noise caused by the data domain shifts. Finally, we propose a distillation method that incorporates explicit features and implicit structured relations to improve the effect of distillation. Experimental results show that our method can quickly converge and obtain higher accuracy than other state-of-the-art methods.

show abstract

Learning Modality-Specific and -Agnostic Representations for Asynchronous Multimodal Language Sequences

Cited by 40 publications

References 23 publications

Recent Trends in 3D Reconstruction of General Non‐Rigid Scenes

Recent Trends in 3D Reconstruction of General Non‐Rigid Scenes

Multimodal automatic assessment of acute pain through facial videos and heart rate signals utilizing transformer-based architectures

Explicit and Implicit Knowledge Distillation via Unlabeled Data

Contact Info

Product

Resources

About