Learning Time Series Counterfactuals via Latent Space Representations

Wang, Zhendong; Samsten, Isak; Mochaourab, Rami; Papapetrou, Panagiotis

doi:10.1007/978-3-030-88942-5_29

Cited by 18 publications

(13 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A related direction has to do with digital human creation [2], [18], [192], as well as with manipulation of human faces [193], [194], [195]. Specifically, a face image of a target individual being superimposed on a video of a source individual has been widely accepted and referred to as deepfake (see Figure 24).…”

Section: Inducing Digital Perturbation Attacksmentioning

confidence: 99%

Synthetic Data in Human Analysis: A Survey

Joshi,

Grimmer,

Rathgeb

et al. 2024

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Deep neural networks have become prevalent in human analysis, boosting the performance of applications, such as biometric recognition, action recognition, as well as person re-identification. However, the performance of such networks scales with the available training data. In human analysis, the demand for large-scale datasets poses a severe challenge, as data collection is tedious, time-expensive, costly and must comply with data protection laws. Current research investigates the generation of synthetic data as an efficient and privacy-ensuring alternative to collecting real data in the field. This survey introduces the basic definitions and methodologies, essential when generating and employing synthetic data for human analysis. We summarise current state-of-the-art methods and the main benefits of using synthetic data. We also provide an overview of publicly available synthetic datasets and generation models. Finally, we discuss limitations, as well as open research problems in this field. This survey is intended for researchers and practitioners in the field of human analysis.

show abstract

Section: Inducing Digital Perturbation Attacksmentioning

confidence: 99%

Synthetic Data in Human Analysis: A Survey

Joshi,

Grimmer,

Rathgeb

et al. 2024

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…Previous face reenactment methods can be divided into warping-based, mesh-based, and NeRF-based. Warpingbased methods (Dong et al 2018;Geng et al 2018;Liu et al 2019;Ha et al 2020;Drobyshev et al 2022;Zhao and Zhang 2022;Wang et al 2022) warp the source features by estimated motion field to transport driving expressions and poses into the source face for 2D generation. Among them, FOMM (Siarohin et al 2019) builds a 2D motion field from the sparse keypoints detected by an unsupervised trained de- itself.…”

Section: One-shot Face Reenactmentmentioning

confidence: 99%

Learning Dense Correspondence for NeRF-Based Face Reenactment

Yang,

Wang,

Lan

et al. 2024

AAAI

View full text Add to dashboard Cite

Face reenactment is challenging due to the need to establish dense correspondence between various face representations for motion transfer. Recent studies have utilized Neural Radiance Field (NeRF) as fundamental representation, which further enhanced the performance of multi-view face reenactment in photo-realism and 3D consistency. However, establishing dense correspondence between different face NeRFs is non-trivial, because implicit representations lack ground-truth correspondence annotations like mesh-based 3D parametric models (e.g., 3DMM) with index-aligned vertexes. Although aligning 3DMM space with NeRF-based face representations can realize motion control, it is sub-optimal for their limited face-only modeling and low identity fidelity. Therefore, we are inspired to ask: Can we learn the dense correspondence between different NeRF-based face representations without a 3D parametric model prior? To address this challenge, we propose a novel framework, which adopts tri-planes as fundamental NeRF representation and decomposes face tri-planes into three components: canonical tri-planes, identity deformations, and motion. In terms of motion control, our key contribution is proposing a Plane Dictionary (PlaneDict) module, which efficiently maps the motion conditions to a linear weighted addition of learnable orthogonal plane bases. To the best of our knowledge, our framework is the first method that achieves one-shot multi-view face reenactment without a 3D parametric model prior. Extensive experiments demonstrate that we produce better results in fine-grained motion control and identity preservation than previous methods.

show abstract

“…[11] extends FOMM for free-view face reenactment. In a new direction, LIA [29] utilizes latent space to make animated face videos.…”

Section: Related Work a Face Reenactmentmentioning

confidence: 99%

F2RPC: Fake to Real Portrait Control From a Virtual Character

Kang,

Kim,

Shim

2023

IEEE Access

View full text Add to dashboard Cite

Existing methods for generating virtual character videos focus on improving either appearance or motion. However, achieving both photo-and motion-realistic characters is critical in real services. To address both aspects, we propose Fake to Real Portrait Control (F2RPC), a unified framework for image destylization and face reenactment. F2RPC employs a blind face restoration model to circumvent GAN inversion limitations, such as identity loss and alignment sensitivity, while preserving GAN's generation quality. This framework includes two novel sub-modules, AdaGPEN for destylization and PCGPEN for reenactment, both leveraging the same restoration model as a backbone. AdaGPEN exploits GAN prior of the restoration model via blending features from the original and its blurred image using the AdaMix block. PCGPEN reenacts the input image to follow the input motion condition via flow-based feature editing. These components function in an end-to-end manner, enhancing efficiency and lowering computational overhead. We evaluate F2RPC using synthetic character dataset and high-resolution talking face datasets for destylization and reenactment, respectively. The results show that F2RPC outperforms the combined use of state-of-the-art methods for destylization (i.e., DualStyleGAN) and reenactment (i.e., StyleHEAT). F2RPC improves the FID by 26.4% and preserves identity similarity by 95% more at a resolution of 512 × 512 video. This evidences F2RPC's efficacy and superiority in the photo-and motion-realistic virtual character video generation task.INDEX TERMS Face morphing, facial reenactment, generative adversarial networks, virtual character destylization.

show abstract

Learning Time Series Counterfactuals via Latent Space Representations

Cited by 18 publications

References 14 publications

Synthetic Data in Human Analysis: A Survey

Synthetic Data in Human Analysis: A Survey

Learning Dense Correspondence for NeRF-Based Face Reenactment

F2RPC: Fake to Real Portrait Control From a Virtual Character

Contact Info

Product

Resources

About